Corbagateway is Unstable and Users are Not Able to Access NMS App (Doc ID 1545301.1)

Last updated on APRIL 10, 2024

Applies to:

Symptoms

Users are unable to login to any NMS apps or users hang on login.

The corbagateway failed, producing a core file. Further stability problems are noticed in the log4j log, and the overall system performance is noticeably worse.

Entries in the log4j log were:

2013-04-10 11:49:37,350 INFO  PublisherBean.counter (                   Logger.java:190)     - srs-output message came early: 315144 expecting: 315125
2013-04-10 11:49:38,154 INFO  PublisherBean.counter (                   Logger.java:190)     - srs-output message came early: 315146 expecting: 315125
2013-04-10 11:49:41,368 INFO  PublisherBean.counter (                   Logger.java:190)     - ejb message came early: 1080192 expecting: 1080175
2013-04-10 11:49:47,781 INFO  PublisherBean.counter (                   Logger.java:190)     - crews message came early: 192804 expecting: 192785
2013-04-10 11:49:47,784 INFO  PublisherBean.counter (                   Logger.java:190)     - srs-output message came early: 315148 expecting: 315125
2013-04-10 11:49:51,787 INFO  PublisherBean.counter (                   Logger.java:190)     - ejb message came early: 1080194 expecting: 1080175
2013-04-10 11:50:02,192 INFO  PublisherBean.counter (                   Logger.java:190)     - ejb message came early: 1080196 expecting: 1080175
2013-04-10 11:50:03,837 INFO  PublisherBean.counter (                   Logger.java:190)     - srs-output message came early: 315150 expecting: 315125

Similar output in the managed server that indicate the same issue could be:

2015-09-11 14:06:34,402 [pool-2-thread-1] INFO com.splwg.oms.ejb.publish.PublisherBean.counter: ejb message came early: 1016 expecting: 1015
2015-09-11 14:06:44,808 [pool-2-thread-1] INFO com.splwg.oms.ejb.publish.PublisherBean.counter: ejb message came early: 1018 expecting: 1015
2015-09-11 14:06:55,217 [pool-2-thread-1] INFO com.splwg.oms.ejb.publish.PublisherBean.counter: ejb message came early: 1020 expecting: 1015
2015-09-11 14:07:05,628 [pool-2-thread-1] INFO com.splwg.oms.ejb.publish.PublisherBean.counter: ejb message came early: 1022 expecting: 1015
2015-09-11 14:07:16,034 [pool-2-thread-1] INFO com.splwg.oms.ejb.publish.PublisherBean.counter: ejb message came early: 1024 expecting: 1015
2015-09-11 14:07:26,443 [pool-2-thread-1] INFO com.splwg.oms.ejb.publish.PublisherBean.counter: ejb message came early: 1026 expecting: 1015

2015-09-11 14:08:34,482 [SwmanListener] ERROR com.splwg.oms.ejb.publish.PublisherBean: Publisher did not received expected message. Resetting connection.

The point to note is that the message counters that were received are consistently not what is expected.

The publisher log also has entries for 2 distinctly different IORs.  These are similar to this

PublishEngine::registerCorbaReceiver - IOR=IOR:000000000000002F49444C3A636F6D2F6365732F696E7465727379735F6167656E742F4

The IOR can be parsed/decoded.

tao_catior can be used on the server running NMS Services.

Put the IOR in a file in the following format: IOR:<IOR String>
Run: tao_catior -f <filename from above line>

For example:
Change directory to the NMS Service logs directory and run the following commands:
ls publisher*log | while read line; do sed -n 's/.*\(IOR:.*\)/\1/p' $line>${line}.IOR; logsave ${line}.ior.out tao_catior -f ${line}.IOR; rm ${line}.IOR; done
ls -rt *.ior.out | while read line; do echo "-----"; echo $line; cat $line|grep -E 'Host|Port'|sort|uniq; done

Review the "Host Name:" and "Port Number:" in the output.

It is highly advised to run a parse on all IOR entries in the log to determine how many different connections are being reference.  In most cases, the IORs that appeared in the publisher log will be found to resolve to be 2 different addresses.

grep can be used in place of sed above: grep -o IOR:.* $line

Changes

Nothing was obviously changed on the system with experiencing the problem.

Cause

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.