Corbagateway is Unstable and Users are Not Able to Access NMS App (Doc ID 1545301.1)

Last updated on NOVEMBER 04, 2016

Applies to:

Oracle Network Management for Utilities - DMS - Version 1.11.0.4 and later
Oracle Utilities Network Management System - Version 1.12.0 and later
Information in this document applies to any platform.

Symptoms

Users are unable to login to any NMS apps or users hang on login.

The corbagateway failed, producing a core file. Further stability problems are noticed in the log4j log, and the overall system performance is noticeably worse.

Entries in the log4j log were:

2013-04-10 11:49:37,350 INFO  PublisherBean.counter (                   Logger.java:190)     - srs-output message came early: 315144 expecting: 315125
2013-04-10 11:49:38,154 INFO  PublisherBean.counter (                   Logger.java:190)     - srs-output message came early: 315146 expecting: 315125
2013-04-10 11:49:41,368 INFO  PublisherBean.counter (                   Logger.java:190)     - ejb message came early: 1080192 expecting: 1080175
2013-04-10 11:49:47,781 INFO  PublisherBean.counter (                   Logger.java:190)     - crews message came early: 192804 expecting: 192785
2013-04-10 11:49:47,784 INFO  PublisherBean.counter (                   Logger.java:190)     - srs-output message came early: 315148 expecting: 315125
2013-04-10 11:49:51,787 INFO  PublisherBean.counter (                   Logger.java:190)     - ejb message came early: 1080194 expecting: 1080175
2013-04-10 11:50:02,192 INFO  PublisherBean.counter (                   Logger.java:190)     - ejb message came early: 1080196 expecting: 1080175
2013-04-10 11:50:03,837 INFO  PublisherBean.counter (                   Logger.java:190)     - srs-output message came early: 315150 expecting: 315125

 

Similar output in the managed server that indicate the same issue could be:

2015-09-11 14:06:34,402 [pool-2-thread-1] INFO com.splwg.oms.ejb.publish.PublisherBean.counter: ejb message came early: 1016 expecting: 1015
2015-09-11 14:06:44,808 [pool-2-thread-1] INFO com.splwg.oms.ejb.publish.PublisherBean.counter: ejb message came early: 1018 expecting: 1015
2015-09-11 14:06:55,217 [pool-2-thread-1] INFO com.splwg.oms.ejb.publish.PublisherBean.counter: ejb message came early: 1020 expecting: 1015
2015-09-11 14:07:05,628 [pool-2-thread-1] INFO com.splwg.oms.ejb.publish.PublisherBean.counter: ejb message came early: 1022 expecting: 1015
2015-09-11 14:07:16,034 [pool-2-thread-1] INFO com.splwg.oms.ejb.publish.PublisherBean.counter: ejb message came early: 1024 expecting: 1015
2015-09-11 14:07:26,443 [pool-2-thread-1] INFO com.splwg.oms.ejb.publish.PublisherBean.counter: ejb message came early: 1026 expecting: 1015

2015-09-11 14:08:34,482 [SwmanListener] ERROR com.splwg.oms.ejb.publish.PublisherBean: Publisher did not received expected message. Resetting connection.

 

The point to note is that the message counters that were received are consistently not what is expected.

 

The publisher log also has entries for 2 distinctly different IORs.  These are similar to this

PublishEngine::registerCorbaReceiver - IOR=IOR:000000000000002F49444C3A636F6D2F6365732F696E7465727379735F6167656E742F4

Note: This is an incomplete IOR entry since a full IOR can be decoded to get an IP address

 The IOR can be parsed/decoded.  One website is http://www2.parc.com/istl/projects/ILU/parseIOR/

It is highly advised to run a parse on all IOR entries in the log to determine how many different connections are being reference.  In most cases, the IORs that appeared in the publisher log will be found to resolve to be 2 different addresses.

Changes

Nothing was obviously changed on the system with experiencing the problem.

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms