Clusterware Cannot Start Normally and Eviction of ASM Instances Occurs When Reconnect Private LAN
(Doc ID 2454243.1)
Last updated on APRIL 17, 2023
Applies to:
Oracle Database - Enterprise Edition - Version 11.2.0.4 to 12.1.0.2 [Release 11.2 to 12.1]Information in this document applies to any platform.
Symptoms
On 12.1.0.2 Grid Infrastructure (GI), when disabled private LAN on one node of 2-nodes RAC and re-enabled it after 10 minutes, CSSD started normally on the evicted node immediately and reconfiguration succeeded.
CRSD(ora.crsd),ASM (ora.asm),HAIP (ora.cluster_interconnect.haip) and other daemon resources also restarted normally, but stopped soon.
In ohasd.trc (under <oracle base>/diag/crs/<hostname>/crs/trace location) of evicted node, below messages were logged when issue occurred.
2018-09-23 16:40:35.178191 : CRSPE:26: {0:3:11} CRS-2676: Start of 'ora.asm' on 'node2' succeeded
2018-09-23 16:40:35.579326 : CRSPE:26: {0:3:11} CRS-2676: Start of 'ora.storage' on 'node2' succeeded
2018-09-23 16:40:36.627164 : CRSPE:26: {0:3:11} CRS-2676: Start of 'ora.crsd' on 'node2' succeeded <--started normmaly
:
:
2018-09-23 16:40:36.634141 : CRSPE:26: {0:3:11} PE Command [ Resource State Change ( ora.crsd 1 1, ) : 6000000002036440 ] has completed
2018-09-23 16:40:36.634417 : AGFW:21: {0:3:11} Agfw Proxy Server received the message: CMD_COMPLETED[Proxy] ID 20482:5048
2018-09-23 16:40:36.634918 : AGFW:21: {0:3:11} Agfw Proxy Server replying to the message: CMD_COMPLETED[Proxy] ID 20482:5048
2018-09-23 16:40:36.635080 : AGFW:21: {0:3:11} Agfw received reply from PE for resource state change for ora.crsd 1 1
2018-09-23 16:40:36.636214 : CRSPE:26: {0:27:3} Placement impossible due to placement policy: no online server passed placement policy filter for [ora.asm 1 1] : 1
2018-09-23 16:40:36.638611 : CRSPE:26: {0:27:3} Placement impossible due to placement policy: no online server passed placement policy filter for [ora.cluster_interconnect.haip 1 1] : 1
2018-09-23 16:40:36.639909 : CRSPE:26: {0:27:3} Placement impossible due to placement policy: no online server passed placement policy filter for [ora.crsd 1 1] : 1
2018-09-23 16:40:36.641207 : CRSPE:26: {0:27:3} Placement impossible due to placement policy: no online server passed placement policy filter for [ora.ctssd 1 1] : 1
2018-09-23 16:40:36.642755 : CRSPE:26: {0:27:3} Placement impossible due to placement policy: no online server passed placement policy filter for [ora.storage 1 1] : 1
2018-09-23 16:40:36.643850 : CRSPE:26: {0:27:3} Operation [STOP of [ora.cssd 1 1] on [node2] : Op:6000000003194440, Cmd:6000000002217630, SeqId:55] has been replaced with [STOP of [ora.cssd 1 1] on [node2] : Op:6000000002f6a1f0, Cmd:6000000002217630, SeqId:0
2018-09-23 16:40:36.644572 : CRSPE:26: {0:27:3} Operation 6000000002f6a1f0 has 15 WOs
2018-09-23 16:40:36.648932 : CRSPE:26: {0:27:3} ICE going for iteration 2 with 8 affected ops
2018-09-23 16:40:36.649472 : CRSPE:26: {0:27:3} Disallowed resource detected:ora.cssd 1 1
2018-09-23 16:40:36.649696 : CRSPE:26: {0:27:3} Disallowed resource detected:ora.cssd 1 1
2018-09-23 16:40:36.650191 : CRSPE:26: {0:27:3} RI [ora.asm 1 1] new internal state: [STOPPING] old value: [STABLE]
2018-09-23 16:40:36.650439 : CRSPE:26: {0:27:3} Sending message to agfw: id = 5049
2018-09-23 16:40:36.650617 : AGFW:21: {0:27:3} Agfw Proxy Server received the message: RESOURCE_STOP[ora.asm 1 1] ID 4099:5049
2018-09-23 16:40:36.650657 : CRSPE:26: {0:27:3} CRS-2673: Attempting to stop 'ora.asm' on 'node2'. <-- stopped soon
:
:
2018-09-23 16:40:36.677600 : CRSPE:26: {0:27:3} CRS-2677: Stop of 'ora.storage' on 'node2' succeeded
2018-09-23 16:40:37.700468 : CRSPE:26: {0:27:3} CRS-2677: Stop of 'ora.crsd' on 'node2' succeeded
2018-09-23 16:40:37.702127 : CRSPE:26: {0:27:3} CRS-2677: Stop of 'ora.ctssd' on 'node2' succeeded
2018-09-23 16:40:37.818396 : CRSPE:26: {0:27:3} CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'node2' succeeded
2018-09-23 16:48:12.226774 : CRSPE:26: {0:27:3} CRS-2677: Stop of 'ora.asm' on 'node2' succeeded
2018-09-23 16:48:13.248118 : CRSPE:26: {0:27:3} CRS-2677: Stop of 'ora.cssd' on 'node2' succeeded
During "shutdown immediate" of ASM instance on node2, "IPC send timeout" was detected because of network interface query failure for HAIP address,and eviction between ASM instances occurred.
As the result, sometimes ASM instance on local node during shutdown will be evicted from cluster. The clusterware on this node cannot start normally again and remains in below status.
Changes
Disconnected private LAN between RAC nodes and re-enabled it after 10 minutes.
Cause
To view full details, sign in with your My Oracle Support account. |
|
Don't have a My Oracle Support account? Click to get started! |
In this Document
| Symptoms |
| Changes |
| Cause |
| Solution |
| References |