CSS Daemon is Terminating With Configuration Mismatch During rootupgrade.sh Execution
(Doc ID 2611611.1)
Last updated on APRIL 17, 2023
Applies to:
Oracle Database - Enterprise Edition - Version 12.2.0.1 and laterInformation in this document applies to any platform.
Symptoms
Upgrading Grid Infrastructure (GI) from 12.2 to 19.3 in a multi-node cluster. rootupgrade successfully completed on node 1 followed by odd numbered nodes. During execution of rootupgrade on remaining even numbered nodes, CSS crash on all nodes except one node.
Surviving node clusterware alert log have following messages
2019-08-05 08:43:24.288 [OCSSD(39025)]CRS-1604: CSSD voting file is offline: o/<IP3>;<IP4>/<DISK PATH>; details at (:CSSNM00069:) in <ORACLE_BASE>/diag/crs/<HOSTNAME3>/crs/trace/ocssd.trc.
2019-08-05 08:43:24.359 [OCSSD(39025)]CRS-1601: CSSD Reconfiguration complete. Active nodes are <HOSTNAME1> <HOSTNAME2> <HOSTNAME3> <HOSTNAME4> <HOSTNAME5> <HOSTNAME6> <HOSTNAME7> <HOSTNAME8> .
2019-08-05 08:43:25.479 [OCTSSD(39287)]CRS-2407: The new Cluster Time Synchronization Service reference node is host <HOSTNAME6>.
2019-08-05 08:43:26.574 [OCSSD(39025)]CRS-1625: Node <HOSTNAME4>, number 4, was shut down
2019-08-05 08:43:27.886 [OCSSD(39025)]CRS-1650: Configuration change 1564994604:0 rejected by node 2 <HOSTNAME2>; details at (:CSSNM000066:) in <ORACLE_BASE>/diag/crs/<HOSTNAME3>/crs/trace/ocssd.trc
2019-08-05 08:43:27.887 [OCSSD(39025)]CRS-1650: Configuration change 1564994604:0 rejected by node 6 <HOSTNAME6>; details at (:CSSNM000066:) in <ORACLE_BASE>/diag/crs/<HOSTNAME3>/crs/trace/ocssd.trc
2019-08-05 08:43:27.888 [OCSSD(39025)]CRS-1650: Configuration change 1564994604:0 rejected by node 8 <HOSTNAME8>; details at (:CSSNM000066:) in <ORACLE_BASE>/diag/crs/<HOSTNAME3>/crs/trace/ocssd.trc
2019-08-05 08:43:24.359 [OCSSD(39025)]CRS-1601: CSSD Reconfiguration complete. Active nodes are <HOSTNAME1> <HOSTNAME2> <HOSTNAME3> <HOSTNAME4> <HOSTNAME5> <HOSTNAME6> <HOSTNAME7> <HOSTNAME8> .
2019-08-05 08:43:25.479 [OCTSSD(39287)]CRS-2407: The new Cluster Time Synchronization Service reference node is host <HOSTNAME6>.
2019-08-05 08:43:26.574 [OCSSD(39025)]CRS-1625: Node <HOSTNAME4>, number 4, was shut down
2019-08-05 08:43:27.886 [OCSSD(39025)]CRS-1650: Configuration change 1564994604:0 rejected by node 2 <HOSTNAME2>; details at (:CSSNM000066:) in <ORACLE_BASE>/diag/crs/<HOSTNAME3>/crs/trace/ocssd.trc
2019-08-05 08:43:27.887 [OCSSD(39025)]CRS-1650: Configuration change 1564994604:0 rejected by node 6 <HOSTNAME6>; details at (:CSSNM000066:) in <ORACLE_BASE>/diag/crs/<HOSTNAME3>/crs/trace/ocssd.trc
2019-08-05 08:43:27.888 [OCSSD(39025)]CRS-1650: Configuration change 1564994604:0 rejected by node 8 <HOSTNAME8>; details at (:CSSNM000066:) in <ORACLE_BASE>/diag/crs/<HOSTNAME3>/crs/trace/ocssd.trc
CSSD trace have following messages
2019-08-05 08:43:27.895 : CSSD:1420224256: [ INFO] clssnmCINUpdateComplete: Committed CIN update completed, config state 4
2019-08-05 08:43:27.895 : CSSD:1437701888: [ INFO] clssnmBldSendUpdate: syncSeqNo(420808201)
2019-08-05 08:43:27.895 : CSSD:1437701888: [ INFO] clssnmBldSendUpdate: using msg version 4
2019-08-05 08:43:27.896 : CSSD:1421801216: [ INFO] clssnmvVoteDiskValidation: Voting disk(o/<IP5>;<IP6>/<DISK PATH>) cluster GUID mismatch
2019-08-05 08:43:27.896 : CSSD:1421801216: [ WARNING] clssnmvWorkerThread: disk o/<IP5>;<IP6>/<DISK PATH> not valid
2019-08-05 08:43:27.896 : CSSD:1421801216: [ INFO] clssnmvDiskAvailabilityChange: voting file o/<IP5>;<IP6>/<DISK PATH> now offline
2019-08-05 08:43:27.896 : CLSF:1425053440: Closing handle:0x7feff00fcd20
2019-08-05 08:43:27.896 : CSSD:1426630400: [ INFO] clssnmvDiskPing: Writing with status 0x3, timestamp 1564994607/12373854
2019-08-05 08:43:27.896 : CLSF:1423378176: Closing handle:0x7feff4085d50
2019-08-05 08:43:27.895 : CSSD:1437701888: [ INFO] clssnmBldSendUpdate: syncSeqNo(420808201)
2019-08-05 08:43:27.895 : CSSD:1437701888: [ INFO] clssnmBldSendUpdate: using msg version 4
2019-08-05 08:43:27.896 : CSSD:1421801216: [ INFO] clssnmvVoteDiskValidation: Voting disk(o/<IP5>;<IP6>/<DISK PATH>) cluster GUID mismatch
2019-08-05 08:43:27.896 : CSSD:1421801216: [ WARNING] clssnmvWorkerThread: disk o/<IP5>;<IP6>/<DISK PATH> not valid
2019-08-05 08:43:27.896 : CSSD:1421801216: [ INFO] clssnmvDiskAvailabilityChange: voting file o/<IP5>;<IP6>/<DISK PATH> now offline
2019-08-05 08:43:27.896 : CLSF:1425053440: Closing handle:0x7feff00fcd20
2019-08-05 08:43:27.896 : CSSD:1426630400: [ INFO] clssnmvDiskPing: Writing with status 0x3, timestamp 1564994607/12373854
2019-08-05 08:43:27.896 : CLSF:1423378176: Closing handle:0x7feff4085d50
On the nodes in which CSSD crashed have following messages
Clusterware alert log
2019-08-05 08:43:27.905 [OCSSD(390880)]CRS-1626: A Configuration change request completed successfully
2019-08-05 08:43:27.905 [OCSSD(390880)]CRS-1642: Node 3 is using a different CSS configuration from the one used by this node; this node is terminating to preserve integrity; details at (:CSSNM00054:) in <ORACLE_BASE>/diag/crs/<HOSTNAME1>/crs/trace/ocssd.trc
2019-08-05 08:43:27.905 [OCSSD(186669)]CRS-1656: The CSS daemon is terminating due to a fatal error; Details at (:CSSSC00012:) in <ORACLE_BASE>/diag/crs/<HOSTNAME1>/crs/trace/ocssd.trc
2019-08-05 08:43:27.905 [OCSSD(390880)]CRS-1642: Node 3 is using a different CSS configuration from the one used by this node; this node is terminating to preserve integrity; details at (:CSSNM00054:) in <ORACLE_BASE>/diag/crs/<HOSTNAME1>/crs/trace/ocssd.trc
2019-08-05 08:43:27.905 [OCSSD(186669)]CRS-1656: The CSS daemon is terminating due to a fatal error; Details at (:CSSSC00012:) in <ORACLE_BASE>/diag/crs/<HOSTNAME1>/crs/trace/ocssd.trc
in cssd trace
2019-08-05 08:43:27.902 : CSSD:2331461376: [ INFO] clssnmvDeconfigureVFs: Changing state for vdisk 0x7f8894229510,o/<IP1>;<IP2>/<DISK PATH>to deconfigured
2019-08-05 08:43:27.902 : CSSD:2331461376: [ INFO] clssnmvDiskStateChange: state from configured to deconfigured disk o/<IP1>;<IP2>/<DISK PATH>
.
. . .
2019-08-05 08:43:27.907 : CSSD:2331461376: [ ERROR] (:CSSNM00054:)clssnmHandleUpdate: Configuration mismatch, mine is CIN 1564994604, unique 0, master has CIN 1564994604, unique 1
2019-08-05 08:43:27.902 : CSSD:2331461376: [ INFO] clssnmvDiskStateChange: state from configured to deconfigured disk o/<IP1>;<IP2>/<DISK PATH>
.
. . .
2019-08-05 08:43:27.907 : CSSD:2331461376: [ ERROR] (:CSSNM00054:)clssnmHandleUpdate: Configuration mismatch, mine is CIN 1564994604, unique 0, master has CIN 1564994604, unique 1
Cause
To view full details, sign in with your My Oracle Support account. |
|
Don't have a My Oracle Support account? Click to get started! |
In this Document
| Symptoms |
| Cause |
| Solution |
| References |