Nokia Siemens NetManager HiS Cartridge Connectivity Is Dropping Intermittently In Production

Last updated on AUGUST 28, 2017

Applies to:

Symptoms

HiS connectivity is dropping intermittently in production. Due to this, business is affected. After restarting ASAP and WL, connection is established to HiS, however, after some time, it is disconnecting again.

The same cartridge code has been working for many years without an issue. Restart of network hardware services and ASAP server was performed. However, the issue persists. While the issue is observed, the telnet connection from ASAP to the NE is working. The cartridge connection is being reconnected sometimes on its own and pending orders get processed until the subsequent occurrence of the issue. There is no specific frequency with which the problem is occurring.

JNEP ERROR:

>> 040628.257:Connection handler viepasade00-28472:LOW :<>:
: (DEBUG) SiNetManagerConnection.sendRequest(): Sent request to NE?: FAIL
>> 040628.257:Connection handler viepasade00-28472:LOW :<>:
: (ERROR) SiNetManagerConnection.sendRequest(): Failed to send request due to connection error: Send request error: Exception waiting for response; nested exception is:
weblogic.utils.net.SocketResetException
>> 040628.257:Connection handler viepasade00-28472:LOW :<>:
com.mslv.activation.cartridge.si.netmanager.x7_0.lnp_entry.qry.QryLnpEntryProcessor: (DEBUG) com.mslv.activation.jinterpreter.ConnectionException: Send request error: Exception waiting for response; nested exception is:
weblogic.utils.net.SocketResetException
at com.mslv.activation.cartridge.si.netmanager.x7_0.SiNetManagerConnection.sendRequest(SiNetManagerConnection.java:500)
at com.mslv.activation.cartridge.si.netmanager.x7_0.lnp_entry.qry.QryLnpEntryProcessor.execute(QryLnpEntryProcessor.java:183)
at com.mslv.activation.cartridge.si.netmanager.x7_0.lnp_entry.qry.generated.QryLnpEntryProxy.execute(QryLnpEntryProxy.java:116)
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.mslv.activation.jinterpreter.JInterpConnection.runClassMethod(JInterpConnection.java:221)
at com.mslv.activation.jinterpreter.JInterpConnection.handleMessage(JInterpConnection.java:131)
at com.mslv.activation.server.ConnectionHandler.run(ConnectionHandler.java:251)

WebLogic Error

<Error> <NETMANAGER_7_0> <viepasade00> <DEp3server> <[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'> <> <> <> <1502454183290> <BEA-000000> <

***** CorbaSender.sendRequest(): Error: Failed to connect to NE with CORBA exception: Exception waiting for response; nested exception is:
              weblogic.utils.net.SocketResetException *****

org.omg.CORBA.MARSHAL: Exception waiting for response; nested exception is:
              weblogic.utils.net.SocketResetException  vmcid: 0x0  minor code: 0  completed: No
              at weblogic.iiop.Utils.mapRemoteToCORBAException(Utils.java:830)
              at weblogic.iiop.Utils.mapToCORBAException(Utils.java:874)
              at weblogic.corba.idl.RemoteDelegateImpl.invoke(RemoteDelegateImpl.java:387)
              at weblogic.corba.idl.RemoteDelegateImpl.invoke(RemoteDelegateImpl.java:341)
              at org.omg.CORBA.portable.ObjectImpl._invoke(ObjectImpl.java:457)
              at enmfos.icn.siemens.DE.NetMFOSMod._NetMFOSIfStub.OpenSession(_NetMFOSIfStub.java:277)
              at com.mslv.activation.cartridge.si.netmanager.x7_0.corba.send.CorbaSender.connect(CorbaSender.java:147)
              at com.mslv.activation.cartridge.si.netmanager.x7_0.async.RequestHandlerBean.onMessage(RequestHandlerBean.java:136)
              at weblogic.ejb.container.internal.MDListener.execute(MDListener.java:583)
              at weblogic.ejb.container.internal.MDListener.transactionalOnMessage(MDListener.java:486)
              at weblogic.ejb.container.internal.MDListener.onMessage(MDListener.java:388)
              at weblogic.jms.client.JMSSession.onMessage(JMSSession.java:4659)
              at weblogic.jms.client.JMSSession.execute(JMSSession.java:4345)
              at weblogic.jms.client.JMSSession.executeMessage(JMSSession.java:3821)

....

<Error> <NETMANAGER_7_0> <viepasade00> <DEp3server> <[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'> <> <> <> <1502454164822> <BEA-000000> <

***** CorbaSender.getReceiverIOR(): Error: Failed to start receiver: Exception waiting for response; nested exception is:
              java.io.EOFException: endOfStream called by muxer *****

org.omg.CORBA.MARSHAL: Exception waiting for response; nested exception is:
              java.io.EOFException: endOfStream called by muxer  vmcid: 0x0  minor code: 0  completed: No
              at weblogic.iiop.Utils.mapRemoteToCORBAException(Utils.java:830)
              at weblogic.iiop.Utils.mapToCORBAException(Utils.java:874)
              at weblogic.corba.idl.RemoteDelegateImpl.invoke(RemoteDelegateImpl.java:387)
              at weblogic.corba.idl.RemoteDelegateImpl.invoke(RemoteDelegateImpl.java:341)
              at org.omg.CORBA.portable.ObjectImpl._invoke(ObjectImpl.java:457)
              at com.mslv.activation.cartridge.si.netmanager.x7_0.corba.receive._CorbaReceiverStub.listen(_CorbaReceiverStub.java:26)
              at com.mslv.activation.cartridge.si.netmanager.x7_0.corba.send.CorbaSender.startReceiver(CorbaSender.java:226)
              at com.mslv.activation.cartridge.si.netmanager.x7_0.corba.send.CorbaSender.connect(CorbaSender.java:169)
              at com.mslv.activation.cartridge.si.netmanager.x7_0.async.RequestHandlerBean.onMessage(RequestHandlerBean.java:136)
              at weblogic.ejb.container.internal.MDListener.execute(MDListener.java:583)
              at weblogic.ejb.container.internal.MDListener.transactionalOnMessage(MDListener.java:486)
              at weblogic.ejb.container.internal.MDListener.onMessage(MDListener.java:388)
              at weblogic.jms.client.JMSSession.onMessage(JMSSession.java:4659)
              at weblogic.jms.client.JMSSession.execute(JMSSession.java:4345)
              at weblogic.jms.client.JMSSession.executeMessage(JMSSession.java:3821)
              at weblogic.jms.client.JMSSession.access$000(JMSSession.java:115)
              at weblogic.jms.client.JMSSession$UseForRunnable.run(JMSSession.java:5170)
              at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:545)
              at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
              at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

Changes

Cause