Bug 27213224 - Deploying Exadata Software Fails At - Step 12 (Initializing Cluster Software) (Doc ID 2391108.1)

root_dbadm02_2018-01-26_16-40-02-663616929.log << dbadm02
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'dbadm02'
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'dbadm02' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
..............
CRS-2672: Attempting to start 'ora.cssd' on 'dbadm02'
CRS-2672: Attempting to start 'ora.diskmon' on 'dbadm02'
CRS-2676: Start of 'ora.diskmon' on 'dbadm02' succeeded
CRS-2883: Resource 'ora.cssdmonitor' failed during Clusterware stack start.
CRS-4406: Oracle High Availability Services synchronous start failed.
CRS-4000: Command Start failed, or completed with errors.
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Status failed, or completed with errors.
2018/01/26 16:52:47 CLSRSC-117: Failed to start Oracle Clusterware stack
Died at /u01/app/12.2.0.1/grid/crs/install/crsinstall.pm line 1494.
The command '/u01/app/12.2.0.1/grid/perl/bin/perl -I/u01/app/12.2.0.1/grid/perl/lib -I/u01/app/12.2.0.1/grid/crs/install /u01/app/12.2.0.1/grid/crs/install/rootcrs.pl ' execution failed

alert.log << dbadm02
~~~~~~~~~

2018-02-01 14:55:25.747 [OCSSD(276600)]CRS-8500: Oracle Clusterware OCSSD process is starting with operating system process ID 276600
2018-02-01 14:55:26.771 [OCSSD(276600)]CRS-1713: CSSD daemon is started in hub mode
2018-02-01 14:57:48.252 [OCSSD(276600)]CRS-1707: Lease acquisition for node dbadm02 number 2 completed
2018-02-01 14:57:49.332 [OCSSD(276600)]CRS-1605: CSSD voting file is online: /dev/exadata_quorum/QD_DBFS_DG_DBADM02; details in /u01/app/grid/diag/crs/dbadm02/crs/trace/ocssd.trc.
2018-02-01 14:57:49.334 [OCSSD(276600)]CRS-1605: CSSD voting file is online: /dev/exadata_quorum/QD_DBFS_DG_DBADM01; details in /u01/app/grid/diag/crs/dbadm02/crs/trace/ocssd.trc.
2018-02-01 14:57:49.339 [OCSSD(276600)]CRS-1605: CSSD voting file is online: o/<IP ADDRESS>/DBFS_DG_CD_05_celadm01; details in /u01/app/grid/diag/crs/dbadm02/crs/trace/ocssd.trc.
2018-02-01 14:57:49.344 [OCSSD(276600)]CRS-1605: CSSD voting file is online: o/<IP ADDRESS>/DBFS_DG_CD_02_celadm02; details in /u01/app/grid/diag/crs/dbadm02/crs/trace/ocssd.trc.
2018-02-01 14:57:49.349 [OCSSD(276600)]CRS-1605: CSSD voting file is online: o/<IP ADDRESS>>/DBFS_DG_CD_06_celadm03; details in /u01/app/grid/diag/crs/dbadm02/crs/trace/ocssd.trc.
2018-02-01 15:00:15.649 [ORAROOTAGENT(275918)]CRS-5818: Aborted command 'check' for resource 'ora.storage'. Details at (:CRSAGF00113:) {0:0:2} in /u01/app/grid/diag/crs/dbadm02/crs/trace/ohasd_orarootagent_root.trc.
2018-02-01 15:01:28.990 [ORAROOTAGENT(288382)]CRS-8500: Oracle Clusterware ORAROOTAGENT process is starting with operating system process ID 288382
2018-02-01 15:05:24.723 [CSSDAGENT(276568)]CRS-5818: Aborted command 'start' for resource 'ora.cssd'. Details at (:CRSAGF00113:) {0:5:3} in /u01/app/grid/diag/crs/dbadm02/crs/trace/ohasd_cssdagent_root.trc.
2018-02-01 15:05:24.779 [OHASD(275755)]CRS-2757: Command 'Start' timed out waiting for response from the resource 'ora.cssd'. Details at (:CRSPE00221:) {0:5:3} in /u01/app/grid/diag/crs/dbadm02/crs/trace/ohasd.trc.
2018-02-01 15:05:25.724 [OCSSD(276600)]CRS-1656: The CSS daemon is terminating due to a fatal error; Details at (:CSSSC00012:) in /u01/app/grid/diag/crs/dbadm02/crs/trace/ocssd.trc
2018-02-01 15:05:25.724 [OCSSD(276600)]CRS-1603: CSSD on node dbadm02 has been shut down.
2018-02-01 15:05:29.009 [ORAROOTAGENT(288382)]CRS-5818: Aborted command 'check' for resource 'ora.storage'. Details at (:CRSAGF00113:) {0:8:2} in /u01/app/grid/diag/crs/dbadm02/crs/trace/ohasd_orarootagent_root.trc.
2018-02-01 15:05:30.724 [OCSSD(276600)]CRS-8503: Oracle Clusterware process OCSSD with operating system process ID 276600 experienced fatal signal or exception code 6.
2018-02-01T15:05:30.728381-02:00
Errors in file /u01/app/grid/diag/crs/dbadm02/crs/trace/ocssd.trc (incident=1):
CRS-8503 [] [] [] [] [] [] [] [] [] [] [] []
Incident details in: /u01/app/grid/diag/crs/dbadm02/crs/incident/incdir_1/ocssd_i1.trc

occsd trc << << dbadm02
~~~~~~~~~~

2018-02-01 14:57:54.806 : CSSD:4267628288: clssnmvDHBValidateNCopy: node 1, dbadm01, has a disk HB, but no network HB, DHB has rcfg 412962613, wrtcnt, 2356, LATS 7737574, lastSeqNo 2351, uniqueness 1517503672, timestamp 1517504273/7015264
2018-02-01 14:57:54.806 : CSSD:4240807680: clssscSelect: gipcwait returned with status gipcretPosted (17)
2018-02-01 14:57:54.806 : CSSD:4240807680: clssnmconnect: connecting to addr gipcha://dbadm01:nm2_x1cluster-clu1
2018-02-01 14:57:54.807 : CSSD:4240807680: clssscConnect: endp 0x15d6 - cookie 0x193cdd0 - addr gipcha://dbadm01:nm2_x1cluster-clu1
2018-02-01 14:57:54.807 : CSSD:4240807680: clssnmconnect: connecting to node(1), endp(0x15d6), flags 0x10002
2018-02-01 14:57:54.807 :GIPCHTHR:4284937984: gipchaWorkerProcessClientConnect: starting resolve from connect for host:dbadm01, port:nm2_x1cluster-clu1, cookie:0x7f8e80047400
2018-02-01 14:57:54.807 :GIPCHDEM:4283361024: gipchaDaemonProcessLookupNameAck: clearing restart flag of node dbadm01
............................
2018-02-01 14:57:55.794 : CSSD:1031788288: clsssc_CLSFAInit_CB: System not ready for CLSFA initialization
2018-02-01 14:57:55.807 : CSSD:4267628288: clssnmvDHBValidateNCopy: node 1, dbadm01, has a disk HB, but no network HB, DHB has rcfg 412962613, wrtcnt, 2361, LATS 7738574, lastSeqNo 2356, uniqueness 1517503672, timestamp 1517504274/7016274
............................
2018-02-01 14:57:56.792 : CSSDGNS:1025480448: clssgnsCrsQuery: Querying CRS for resource type "ora.gns.type".
2018-02-01 14:57:56.793 :GIPCXCPT:1025480448: gipcInternalSetAttribute: failed during gipcInternalSetAttribute, ret gipcretInvalidAttribute (5)
2018-02-01 14:57:56.793 :GIPCXCPT:1025480448: gipcSetAttributeNativeF [clscrsconGipcConnect : clscrscon.c : 655]: EXCEPTION[ ret gipcretInvalidAttribute (5) ] failure for obj 0x7f8f1815e3c0 [000000000000163b] { gipcEndpoint : localAddr '', remoteAddr '', numPend 0, numReady 0, numDone 0, numDead 0, numTransfer 0, objFlags 0x0, pidPeer 0, readyRef (nil), ready 0, wobj (nil), sendp (nil) status 13flags 0x20000000, flags-2 0x0, usrFlags 0x0 }, name 'traceLevel', val 0x7f8f3d1e9554, len 4, flags 0x0
2018-02-01 14:57:56.793 :GIPCXCPT:1025480448: gipcInternalConnectSync: failed sync request, ret gipcretConnectionRefused (29)

occsd.trc <<<< dbadm01
~~~~~~~~~~~~~~~~~~~~~~~~~~
2018-02-01 14:57:49.414 :GIPCHAUP:1109665536: gipchaUpperAccept: completed accept endp 0x7f67fc0b1e40 [000000000000e0b2] { gipchaEndpoint : port 'nm2_x1cluster-clu1/75de-a21b-baf6-6504', peer 'dbadm02:a05e-a3a4-8510-f400', srcCid 00000000-0000e0b2, dstCid 00000000-000012be, numSend 0, maxSend 100, groupListType 1, hagroup 0x2ecbf80, priority 0, forceAckCount 0, usrFlags 0x4000, flags 0x204 }
2018-02-01 14:57:49.414 : CSSD:3345979136: clssnmeventhndlr: gipcAssociate endp 0xe0bd in container 0x255 type of conn gipcha
2018-02-01 14:57:49.418 : CSSD:3345979136: clssnmConnComplete: Rejecting connection from node 2 as MultiNode RAC is not supported or certified in this Configuration
2018-02-01 14:57:49.418 :GIPCGMOD:3345979136: gipcmodGipcDisconnect: [gipc] Issued endpoint close for endp 0x7f67fc0b26a0 [000000000000e0bd] { gipcEndpoint : localAddr 'gipcha://dbadm01:nm2_x1cluster-clu1/75de-a21b-baf6-6504', remoteAddr 'gipcha://dbadm02:a05e-a3a4-8510-f400', numPend 1, numReady 0, numDone 0, numDead 0, numTransfer 0, objFlags 0x0, pidPeer 0, readyRef 0x2ecb6c0, ready 0, wobj 0x7f67fc060d50, sendp (nil) status 0flags 0x26138606, flags-2 0x0, usrFlags 0x0 }

-bash-4.4$ ggrep "Rejecting connection from node" ocssd.trc
2018-02-01 14:57:49.418 : CSSD:3345979136: clssnmConnComplete: Rejecting connection from node 2 as MultiNode RAC is not supported or certified in this Configuration
2018-02-01 14:57:51.766 : CSSD:3345979136: clssnmConnComplete: Rejecting connection from node 2 as MultiNode RAC is not supported or certified in this Configuration
2018-02-01 14:57:54.767 : CSSD:3345979136: clssnmConnComplete: Rejecting connection from node 2 as MultiNode RAC is not supported or certified in this Configuration
2018-02-01 14:57:57.769 : CSSD:3345979136: clssnmConnComplete: Rejecting connection from node 2 as MultiNode RAC is not supported or certified in this Configuration
2018-02-01 14:58:00.771 : CSSD:3345979136: clssnmConnComplete: Rejecting connection from node 2 as MultiNode RAC is not supported or certified in this Configuration
2018-02-01 14:58:03.774 : CSSD:3345979136: clssnmConnComplete: Rejecting connection from node 2 as MultiNode RAC is not supported or certified in this Configuration
...
2018-02-01 15:05:15.135 : CSSD:3345979136: clssnmConnComplete: Rejecting connection from node 2 as MultiNode RAC is not supported or certified in this Configuration
2018-02-01 15:05:18.138 : CSSD:3345979136: clssnmConnComplete: Rejecting connection from node 2 as MultiNode RAC is not supported or certified in this Configuration
2018-02-01 15:05:21.141 : CSSD:3345979136: clssnmConnComplete: Rejecting connection from node 2 as MultiNode RAC is not supported or certified in this Configuration
2018-02-01 15:05:24.143 : CSSD:3345979136: clssnmConnComplete: Rejecting connection from node 2 as MultiNode RAC is not supported or certified in this Configuration