12.1.0.1 GI Fails To Start After Broken Private Interconnect Is Restored

(Doc ID 1901627.1)

Last updated on AUGUST 07, 2014

Applies to:

Oracle Database - Enterprise Edition - Version 12.1.0.1 to 12.1.0.1 [Release 12.1]
Information in this document applies to any platform.

Symptoms

After private network is restored, GI stack does not come up successfully:

From ocssd.log
==========

2014-06-19 06:27:04.651: [ CSSD][3101423360]clssscthrdmain: Terminating thread clssnmSendingThread
2014-06-19 06:27:04.718: [ CSSD][3106162432]clssnmvDHBValidateNCopy: node 1, racnode1, has a disk HB, but no network HB, DHB has rcfg 298108266, wrtcnt, 10424132, LATS 504432874, lastSeqNo 10424131, uniqueness 1402668773, timestamp 1403173624/503901794


// bond1 is disabled on node1 at 6:26


2014-06-19 06:27:04.718: [ CSSD][3099838208]clssnmSendSync: syncSeqNo(298108266), indicating EXADATA fence initialization complete
2014-06-19 06:27:04.718: [ CSSD][3099838208]List of nodes that have ACKed my sync: NULL
2014-06-19 06:27:07.192: [ CSSD][757114432](TLM) Starting CSS daemon, version 12.1.0.1.0 with uniqueness value 1403173627
2014-06-19 06:27:07.192: [ CSSD][757114432]clsu_load_ENV_levels: Module = CSSD, LogLevel = 2, TraceLevel = 0

...

2014-06-19 06:39:49.692: [ CSSD][1091962624]clssnmRcfgMgrThread: Local Join
2014-06-19 06:39:49.692: [ CSSD][1091962624]clssnmLocalJoinEvent: begin on node(2), waittime 193000
2014-06-19 06:39:49.692: [ CSSD][1091962624]clssnmLocalJoinEvent: set curtime (505197734) for my node
2014-06-19 06:39:49.692: [ CSSD][1091962624]clssnmLocalJoinEvent: scanning 32 nodes
2014-06-19 06:39:49.692: [ CSSD][1091962624]clssnmLocalJoinEvent: Node racnode1, number 1, is in an existing cluster with disk state 3

...

2014-06-19 06:47:19.118: [ CSSD][1140365056]clssscagProcAgReq: shutdown abort requested by the agent
2014-06-19 06:47:19.118: [ CSSD][1140365056]###################################
2014-06-19 06:47:19.118: [ CSSD][1140365056]clssscExit: CSSD aborting from thread clssscAgListener
2014-06-19 06:47:19.118: [ CSSD][1140365056]###################################
2014-06-19 06:47:19.118: [ CSSD][1140365056](:CSSSC00012:)clssscExit: A fatal error occurred and the CSS daemon is terminating abnormally

...

2014-06-19 06:47:24.295: [GIPCHALO][1115092736]gipchaLowerProcessNode: no valid interfaces found to node for 6000 ms, node 0x7f0f08190c40 { host 'racnode1', haName 'aff5-71d5-2d19-3e9c', srcLuid 1b5d6dfb-ed2da8c5, dstLuid 00000000-00000000 numInf 0, sentRegister 0, localMonitor 0, baseStream 0x7f0f0818f740 type gipchaNodeType12001 (20), nodeIncarnation 4632ed0b-fff23a8c incarnation 2 flags 0x100004}
2014-06-19 06:47:32.972: [ CSSD][2415949376](TLM) Starting CSS daemon, version 12.1.0.1.0 with uniqueness value 1403174852
2014-06-19 06:47:32.972: [ CSSD][2415949376]clsu_load_ENV_levels: Module = CSSD, LogLevel = 2, TraceLevel = 0

...

2014-06-19 06:57:31.718: [ CSSD][2358707968]clssnmLocalJoinEvent: takeover aborted due to cluster member node found on disk
2014-06-19 06:57:31.784: [GIPCHALO][2381842176]gipchaLowerSendEstablish: sending establish message for node '0x7fc954190c40 { host 'racnode1', haName 'aff5-71d5-2d19-3e9c', srcLuid 9f4e8eda-4ad95f0f, dstLuid 00000000-00000000 numInf 0, sentRegister 0, localMonitor 0, baseStream 0x7fc954185180 type gipchaNodeType12001 (20), nodeIncarnation 4632ed0b-fff23a8c incarnation 2 flags 0x100004}'
2014-06-19 06:57:32.034: [ CSSD][2407114496]clssscagProcAgReq: shutdown abort requested by the agent
2014-06-19 06:57:32.034: [ CSSD][2407114496]###################################
2014-06-19 06:57:32.034: [ CSSD][2407114496]clssscExit: CSSD aborting from thread clssscAgListener
2014-06-19 06:57:32.034: [ CSSD][2407114496]###################################
2014-06-19 06:57:32.034: [ CSSD][2407114496](:CSSSC00012:)clssscExit: A fatal error occurred and the CSS daemon is terminating abnormally


2014-06-19 06:57:32.034: [ CSSD][2407114496]### Begin diagnostic data for the NM layer ###
2014-06-19 06:57:32.034: [ CSSD][2407114496]Local node racnode2, number 2, state is clssnmNodeStateJOINING
2014-06-19 06:57:32.034: [ CSSD][2407114496]Status for node racnode1, number 1, uniqueness 1402668773, node ID 0 2014-06-19 06:57:32.034: [ CSSD][2407114496] State clssnmNodeStateINACTIVE, Connect: started 1 completed 0 OK
2014-06-19 06:57:32.034: [ CSSD][2407114496]Status for node racnode2, number 2, uniqueness 1403174852, node ID 0
2014-06-19 06:57:32.034: [ CSSD][2407114496] State clssnmNodeStateJOINING, Connect: started 1 completed 1 OK

 

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms