My Oracle Support Banner

CSS Daemon is Terminating With Configuration Mismatch During rootupgrade.sh Execution (Doc ID 2611611.1)

Last updated on NOVEMBER 18, 2019

Applies to:

Oracle Database - Enterprise Edition - Version 12.2.0.1 and later
Information in this document applies to any platform.

Symptoms

Upgrading Grid Infrastructure (GI) from 12.2 to 19.3 in a multi-node cluster. rootupgrade successfully completed on node 1 followed by odd numbered nodes. During execution of rootupgrade on remaining even numbered nodes, CSS crash on all nodes except one node.

Surviving node clusterware alert log have following messages

2019-08-05 08:43:24.288 [OCSSD(39025)]CRS-1604: CSSD voting file is offline: o/<IP3>;<IP4>/<DISK PATH>; details at (:CSSNM00069:) in <ORACLE_BASE>/diag/crs/<HOSTNAME3>/crs/trace/ocssd.trc.
2019-08-05 08:43:24.359 [OCSSD(39025)]CRS-1601: CSSD Reconfiguration complete. Active nodes are <HOSTNAME1> <HOSTNAME2> <HOSTNAME3> <HOSTNAME4> <HOSTNAME5> <HOSTNAME6> <HOSTNAME7> <HOSTNAME8> .
2019-08-05 08:43:25.479 [OCTSSD(39287)]CRS-2407: The new Cluster Time Synchronization Service reference node is host <HOSTNAME6>.
2019-08-05 08:43:26.574 [OCSSD(39025)]CRS-1625: Node <HOSTNAME4>, number 4, was shut down
2019-08-05 08:43:27.886 [OCSSD(39025)]CRS-1650: Configuration change 1564994604:0 rejected by node 2 <HOSTNAME2>; details at (:CSSNM000066:) in <ORACLE_BASE>/diag/crs/<HOSTNAME3>/crs/trace/ocssd.trc
2019-08-05 08:43:27.887 [OCSSD(39025)]CRS-1650: Configuration change 1564994604:0 rejected by node 6 <HOSTNAME6>; details at (:CSSNM000066:) in <ORACLE_BASE>/diag/crs/<HOSTNAME3>/crs/trace/ocssd.trc
2019-08-05 08:43:27.888 [OCSSD(39025)]CRS-1650: Configuration change 1564994604:0 rejected by node 8 <HOSTNAME8>; details at (:CSSNM000066:) in <ORACLE_BASE>/diag/crs/<HOSTNAME3>/crs/trace/ocssd.trc



CSSD trace have following messages

2019-08-05 08:43:27.895 :    CSSD:1420224256: [     INFO] clssnmCINUpdateComplete: Committed CIN update completed, config state 4
2019-08-05 08:43:27.895 :    CSSD:1437701888: [     INFO] clssnmBldSendUpdate: syncSeqNo(420808201)
2019-08-05 08:43:27.895 :    CSSD:1437701888: [     INFO] clssnmBldSendUpdate: using msg version 4
2019-08-05 08:43:27.896 :    CSSD:1421801216: [     INFO] clssnmvVoteDiskValidation: Voting disk(o/<IP5>;<IP6>/<DISK PATH>) cluster GUID mismatch
2019-08-05 08:43:27.896 :    CSSD:1421801216: [  WARNING] clssnmvWorkerThread: disk o/<IP5>;<IP6>/<DISK PATH> not valid
2019-08-05 08:43:27.896 :    CSSD:1421801216: [     INFO] clssnmvDiskAvailabilityChange: voting file o/<IP5>;<IP6>/<DISK PATH> now offline
2019-08-05 08:43:27.896 :    CLSF:1425053440: Closing handle:0x7feff00fcd20
2019-08-05 08:43:27.896 :    CSSD:1426630400: [     INFO] clssnmvDiskPing: Writing with status 0x3, timestamp 1564994607/12373854
2019-08-05 08:43:27.896 :    CLSF:1423378176: Closing handle:0x7feff4085d50


On the nodes in which CSSD crashed have following messages

Clusterware alert log

 

2019-08-05 08:43:27.905 [OCSSD(390880)]CRS-1626: A Configuration change request completed successfully
2019-08-05 08:43:27.905 [OCSSD(390880)]CRS-1642: Node 3 is using a different CSS configuration from the one used by this node; this node is terminating to preserve integrity; details at (:CSSNM00054:) in <ORACLE_BASE>/diag/crs/<HOSTNAME1>/crs/trace/ocssd.trc
2019-08-05 08:43:27.905 [OCSSD(186669)]CRS-1656: The CSS daemon is terminating due to a fatal error; Details at (:CSSSC00012:) in <ORACLE_BASE>/diag/crs/<HOSTNAME1>/crs/trace/ocssd.trc



in cssd trace

2019-08-05 08:43:27.902 :    CSSD:2331461376: [     INFO] clssnmvDeconfigureVFs: Changing state for vdisk 0x7f8894229510,o/<IP1>;<IP2>/<DISK PATH>to deconfigured
2019-08-05 08:43:27.902 :    CSSD:2331461376: [     INFO] clssnmvDiskStateChange: state from configured to deconfigured disk o/<IP1>;<IP2>/<DISK PATH>
.
. . .
2019-08-05 08:43:27.907 :    CSSD:2331461376: [    ERROR] (:CSSNM00054:)clssnmHandleUpdate: Configuration mismatch, mine is CIN 1564994604, unique 0, master has CIN 1564994604, unique 1 

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


In this Document
Symptoms
Cause
Solution


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.