InfoCenter Server will not Restart when Other Instances have Jgroup Errors

(Doc ID 2092381.1)

Last updated on MAY 24, 2017

Applies to:

Oracle Knowledge - Version 8.6 and later
Information in this document applies to any platform.

Symptoms

IC systems are getting jgroup errors and 7 ICs are down and cannot restart.

4211807257 [Timer-3,InquiraEOFGroupPROD,ok74kmwp0000006-2276] ERROR org.jgroups.util.TimeScheduler2  - failed running task FD_ALL: TimeoutChecker (interval=2000 ms)
java.lang.NoClassDefFoundError: org/jgroups/protocols/VERIFY_SUSPECT$Entry
at org.jgroups.protocols.VERIFY_SUSPECT.addSuspect(VERIFY_SUSPECT.java:246)
at org.jgroups.protocols.VERIFY_SUSPECT.verifySuspect(VERIFY_SUSPECT.java:196)
at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:104)
at org.jgroups.protocols.FD_ALL.suspect(FD_ALL.java:347)
at org.jgroups.protocols.FD_ALL$TimeoutChecker.run(FD_ALL.java:422)
at org.jgroups.util.TimeScheduler2$RecurringTask.run(TimeScheduler2.java:561)
at org.jgroups.util.TimeScheduler2$MyTask.run(TimeScheduler2.java:493)
at org.jgroups.util.TimeScheduler2$Entry.execute(TimeScheduler2.java:398)
at org.jgroups.util.TimeScheduler2$1.run(TimeScheduler2.java:258)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
4211807595 [Timer-5,InquiraEOFGroupPROD,ok74kmwp0000006-2276] ERROR org.jgroups.util.TimeScheduler2  - failed running task NAKACK2: RetransmitTask (interval=1000 ms)
java.lang.NoClassDefFoundError: org/jgroups/util/Table$Missing
at org.jgroups.util.Table.getMissing(Table.java:572)
at org.jgroups.protocols.pbcast.NAKACK2.triggerXmit(NAKACK2.java:1424)
at org.jgroups.protocols.pbcast.NAKACK2$RetransmitTask.run(NAKACK2.java:1408)
at org.jgroups.util.TimeScheduler2$RecurringTask.run(TimeScheduler2.java:561)
at org.jgroups.util.TimeScheduler2$MyTask.run(TimeScheduler2.java:493)
at org.jgroups.util.TimeScheduler2$Entry.execute(TimeScheduler2.java:398)
at org.jgroups.util.TimeScheduler2$1.run(TimeScheduler2.java:258)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

3 went down on sunday so those were tried to restart in various ways bringing down the whole server (6) instances in trying to restart the managed server. these are all on the 0002 box.

When restarting the instances they cannot move pass the jgroups startup and appear to be hung.

Log 0002 where it stops.
Property name=varchar50
Property type=VARCHAR2
Property name=varchar500
Property type=VARCHAR2
2276 [ProcessChangesQueue] WARN org.jgroups.stack.Configurator  - JGRP000014: TP.discard_incompatible_packets has been deprecated: incompatible packets are discarded anyway
2277 [ProcessChangesQueue] WARN org.jgroups.stack.Configurator  - JGRP000014: TP.enable_bundling has been deprecated: will be ignored as bundling is on by default
2307 [ProcessChangesQueue] WARN org.jgroups.stack.Configurator  - JGRP000014: UNICAST.setTimeout has been deprecated: not used anymore

-------------------------------------------------------------------
GMS: address=ok74kmwp0000002-24308, cluster=InquiraEOFGroupPROD, physical address=10.96.42.63:57221
-------------------------------------------------------------------
[GC (Allocation Failure)  1317620K->171959K(4121600K), 0.1048264 secs]
[GC (Allocation Failure)  1247159K->111739K(4134400K), 0.0460476 secs]
[GC (Allocation Failure)  1203323K->106893K(4125184K), 0.0456545 secs]
[GC (Allocation Failure)  1198477K->107058K(4148224K), 0.0315748 secs]
[GC (Allocation Failure)  1228850K->107118K(4141568K), 0.0425406 secs]
[GC (Allocation Failure)  1228910K->107182K(4159488K), 0.0518262 secs]
[GC (Allocation Failure)  1252526K->112022K(4153856K), 0.0667649 secs]
[GC (Allocation Failure)  1257366K->107186K(4170240K), 0.0461792 secs]

Changes

When a new jgroups name is given the instances start right up.

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms