ODI J2EE Agent Fails to Start Because it Fails to Join the Coherence Cluster Raising "Time out exception and The service timed out due to unanswered handshake request"

Last updated on JULY 11, 2017

Applies to:

Symptoms

Oracle Data Integrator (ODI) J2EE Agent fails to start because it is unable to join the configured Coherence Cluster.

Generated error:

Time out exception and The service timed out due to unanswered handshake request

The Weblogic Server logs is showing the following errors:

<Error> <com.oracle.coherence> <clssitsoa1.eis.local> <ODI_server1> <Logger@ 12.1.3.0.0> <> <> <9846971f-1b21-45af-a4a3-3c71295acadb-00000001> <1440835094731> <BEA-000000> <2015-08-29 10:58:14.731/421.056 Oracle Coherence GE 12.1.3.0.0 <Error> (thread=[STANDBY] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)', member=5): Error while starting service "Management": com.tangosol.net.RequestTimeoutException: Timeout during service start: ServiceInfo(Id=2, Name=Management, Type=Invocation

 MemberSet=ServiceMemberSet(
   OldestMember=Member(Id=1, Timestamp=2015-08-29 10:53:02.527, Address=10.70.1.28:9991, MachineId=52574, Location=site:,machine:clssitsoa1.eis.local,process:30305,member:osb_server1, Role=WeblogicServer)
   ActualMemberSet=MemberSet(Size=6
     Member(Id=1, Timestamp=2015-08-29 10:53:02.527, Address=10.70.1.28:9991, MachineId=52574, Location=site:,machine:clssitsoa1.eis.local,process:30305,member:osb_server1, Role=WeblogicServer)
     Member(Id=2, Timestamp=2015-08-29 10:53:06.062, Address=10.70.1.27:9991, MachineId=32799, Location=site:,machine:clssitsoa2.eis.local,process:17503,member:soa_server2, Role=WeblogicServer)
     Member(Id=3, Timestamp=2015-08-29 10:53:06.105, Address=10.70.1.29:9991, MachineId=32799, Location=site:,machine:clssitsoa2.eis.local,process:17420,member:osb_server2, Role=WeblogicServer)
     Member(Id=4, Timestamp=2015-08-29 10:53:09.028, Address=10.70.1.26:9991, MachineId=52574, Location=site:,machine:clssitsoa1.eis.local,process:30336,member:soa_server1, Role=WeblogicServer)
     Member(Id=5, Timestamp=2015-08-29 10:53:13.962, Address=10.70.1.37:9991, MachineId=52574, Location=site:,machine:clssitsoa1.eis.local,process:30322,member:ODI_server1, Role=WeblogicServer)
     Member(Id=7, Timestamp=2015-08-29 10:53:22.673, Address=10.70.1.24:9991, MachineId=52574, Location=site:,machine:clssitsoa1.eis.local,process:30483,member:WLS_WSM1, Role=WeblogicServer)
     )
   MemberId|ServiceVersion|ServiceJoined|MemberState|EndPoint
     1|12.1.3|2015-08-29 10:53:06.094|JOINED|shared,
     2|12.1.3|2015-08-29 10:53:06.453|JOINED|shared,
     3|12.1.3|2015-08-29 10:53:06.66|JOINED|shared,
     4|12.1.3|2015-08-29 10:53:09.725|JOINED|shared,
     5|12.1.3|2015-08-29 10:53:14.604|JOINING|shared,
     7|12.1.3|2015-08-29 10:58:11.697|JOINING|shared
   )
)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onStartupTimeout(Grid.CDB:3)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.start(Service.CDB:28)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.start(Grid.CDB:6)
at com.tangosol.coherence.component.util.SafeService.startService(SafeService.CDB:57)
at com.tangosol.coherence.component.util.SafeService.ensureRunningService(SafeService.CDB:27)
at com.tangosol.coherence.component.util.SafeService.start(SafeService.CDB:14)
at com.tangosol.coherence.component.net.management.Connector.startService(Connector.CDB:72)
at com.tangosol.coherence.component.net.management.gateway.Remote.registerLocalModel(Remote.CDB:10)
at com.tangosol.coherence.component.net.management.gateway.Local.registerLocalModel(Local.CDB:11)
at com.tangosol.coherence.component.net.management.Gateway.register(Gateway.CDB:6)
at com.tangosol.coherence.component.util.SafeCluster.ensureRunningCluster(SafeCluster.CDB:53)
at com.tangosol.coherence.component.util.SafeCluster.start(SafeCluster.CDB:2)
at com.tangosol.net.CacheFactory.ensureCluster(CacheFactory.java:428)
at weblogic.coherence.service.internal.WLSCoherenceConfigurator.configureCoherence(WLSCoherenceConfigurator.java:146)
at ...

 And another error:

weblogic.cacheprovider.coherence.CoherenceException: Failed to start Service "Cluster" (ServiceState=SERVICE_STOPPED, STATE_JOINED)
 at weblogic.cacheprovider.coherence.CoherenceClusterManager.processException(CoherenceClusterManager.java:811)
 at weblogic.cacheprovider.coherence.CoherenceClusterManager.configureClusterService(CoherenceClusterManager.java:250)
 at weblogic.cacheprovider.CacheProviderServerService.bootCoherenceFromWLSCluster(CacheProviderServerService.java:225)
 at weblogic.cacheprovider.CacheProviderServerService.initCoherence(CacheProviderServerService.java:94)
 at weblogic.cacheprovider.CacheProviderServerService.initialize(CacheProviderServerService.java:71)
 at weblogic.cacheprovider.CacheProviderServerService.start(CacheProviderServerService.java:65)
 at weblogic.server.AbstractServerService.postConstruct(AbstractServerService.java:78)
 at ...
Caused by: java.lang.RuntimeException: Failed to start Service "Cluster" (ServiceState=SERVICE_STOPPED, STATE_JOINED)
 at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.start(Service.CDB:38)
 at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.start(Grid.CDB:6)
 at com.tangosol.coherence.component.net.Cluster.startSystemServices(Cluster.CDB:4)
 at com.tangosol.coherence.component.net.Cluster.onStart(Cluster.CDB:53)
 at com.tangosol.coherence.component.net.Cluster.start(Cluster.CDB:11)
 at com.tangosol.coherence.component.util.SafeCluster.startCluster(SafeCluster.CDB:4)
 at com.tangosol.coherence.component.util.SafeCluster.restartCluster(SafeCluster.CDB:10)
 at com.tangosol.coherence.component.util.SafeCluster.ensureRunningCluster(SafeCluster.CDB:30)
 at com.tangosol.coherence.component.util.SafeCluster.start(SafeCluster.CDB:2)
 at com.tangosol.net.CacheFactory.ensureCluster(CacheFactory.java:428)
 at weblogic.coherence.service.internal.WLSCoherenceConfigurator.configureCoherence(WLSCoherenceConfigurator.java:146)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:606)
 at weblogic.cacheprovider.coherence.CoherenceClusterManager.configureClusterService(CoherenceClusterManager.java:246)
 ... 22 more 

Changes

Cause