Coherence error "Delaying formation of a new cluster; waiting for well-known nodes to respond" after network issue prevented SOA server start up (Doc ID 2056055.1)

Last updated on NOVEMBER 07, 2016

Applies to:

Oracle SOA Suite - Version 11.1.1.7.0 and later
Information in this document applies to any platform.

Symptoms

Due to a network issue in the machines where the SOA and Coherence servers are running, the servers were restarted.

After that, the Admin and Managed SOA servers are unable to start up again. Looking into the log files, the following messages are seen:

<Sep 14, 2015 11:16:06 AM CEST> <Warning> <Coherence> <BEA-000000> <2015-09-14 11:16:06.584/997.564 Oracle Coherence GE 3.7.1.1 <Warning> (thread=Cluster, member=n/a): Delaying formation of a new cluster; waiting for well-known nodes to respond>
<Sep 14, 2015 11:16:12 AM CEST> <Error> <Coherence> <BEA-000000> <2015-09-14 11:16:12.306/1003.286 Oracle Coherence GE 3.7.1.1 <Error> (thread=Cluster, member=n/a): Detected soft timeout) of {WrapperGuardable Guard{Daemon=IpMonitor} Service=ClusterService{Name=Cluster, State=(SERVICE_STARTED, STATE_ANNOUNCE), Id=0, Version=3.7.1}}>
<Sep 14, 2015 11:16:12 AM CEST> <Error> <Coherence> <BEA-000000> <2015-09-14 11:16:12.361/1003.341 Oracle Coherence GE 3.7.1.1 <Error> (thread=Recovery Thread, member=n/a): Full Thread Dump
...
<Sep 14, 2015 11:12:08 AM CEST> <Warning> <Coherence> <BEA-000000> <2015-09-14 11:12:08.298/759.278 Oracle Coherence GE 3.7.1.1 <Warning> (thread=Cluster, member=n/a): This Member(Id=0, Timestamp=2015-09-14 11:11:37.797, Address=10.0.0.101:8888, MachineId=1001, Location=site:,machine:SOA_VIP101,process:4626, Role=WeblogicServer) has been attempting to join the cluster using WKA list [<list>] for 30 seconds without success; this could indicate a mis-configured WKA, or it may simply be the result of a busy cluster or active failover.>
<Sep 14, 2015 11:12:08 AM CEST> <Warning> <Coherence> <BEA-000000> <2015-09-14 11:12:08.298/759.278 Oracle Coherence GE 3.7.1.1 <Warning> (thread=Cluster, member=n/a): Delaying formation of a new cluster; waiting for well-known nodes to respond>
<Sep 14, 2015 11:12:15 AM CEST> <Warning> <Coherence> <BEA-000000> <2015-09-14 11:12:15.106/766.086 Oracle Coherence GE 3.7.1.1 <Warning> (thread=Cluster, member=n/a): Delaying formation of a new cluster; waiting for well-known nodes to respond>
<Sep 14, 2015 11:12:21 AM CEST> <Warning> <Coherence> <BEA-000000> <2015-09-14 11:12:21.915/772.895 Oracle Coherence GE 3.7.1.1 <Warning> (thread=Cluster, member=n/a): Delaying formation of a new cluster; waiting for well-known nodes to respond>
...
<Sep 14, 2015 11:16:38 AM CEST> <Error> <oracle.integration.platform.blocks.deploy.coordinator> <BEA-000000> <Cluster communication initialization failed.  If you are using multicast, please make sure multicast is enabled on your network and that there is no interference on the address in use.  Please see the documentation for more details.
com.tangosol.net.RequestTimeoutException: Timeout during service start: ServiceInfo(Id=0, Name=Cluster, Type=Cluster
  MemberSet=MasterMemberSet(
    ThisMember=null
    OldestMember=null
    ActualMemberSet=MemberSet(Size=0
      )
    MemberId|ServiceVersion|ServiceJoined|MemberState
    RecycleMillis=1200000
    RecycleSet=MemberSet(Size=0
      )
    )
)
    at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onStartupTimeout(Grid.CDB:3)
    at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.start(Service.CDB:28)
    at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.start(Grid.CDB:6)
    at com.tangosol.coherence.component.net.Cluster.onStart(Cluster.CDB:56)
    at com.tangosol.coherence.component.net.Cluster.start(Cluster.CDB:11)
    at com.tangosol.coherence.component.util.SafeCluster.startCluster(SafeCluster.CDB:3)
    at com.tangosol.coherence.component.util.SafeCluster.restartCluster(SafeCluster.CDB:10)
    at com.tangosol.coherence.component.util.SafeCluster.ensureRunningCluster(SafeCluster.CDB:26)
    at com.tangosol.coherence.component.util.SafeCluster.start(SafeCluster.CDB:2)
    at com.tangosol.net.CacheFactory.ensureCluster(CacheFactory.java:427)



Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms