OCI DBCS : Failed to start CRS on first RAC node - (GIPC) failed to identify the Fast Node Death Detection (FNDD). (Doc ID 2969313.1)

Last updated on OCTOBER 06, 2023

Applies to:

Symptoms

CRS failed to start on first node of two node RAC

CRS alert.log in NODE1
=====

2023-07-25 09:15:36.871 [GIPCD(37351)]CRS-8500: Oracle Clusterware GIPCD process is starting with operating system process ID 37351
2023-07-25 09:15:38.623 [GIPCD(37351)]CRS-7517: The Oracle Grid Interprocess Communication (GIPC) failed to identify the Fast Node Death Detection (FNDD).

2023-07-25 09:15:51.721 [OCSSD(37536)]CRS-1621: The IPMI configuration data for this node stored in the Oracle registry is incomplete; details at (:CSSNK00002:) in /u01/app/grid/diag/crs/NODE1/crs/trace/ocssd.trc
2023-07-25 09:15:51.722 [OCSSD(37536)]CRS-1617: The information required to do node kill for node NODE1 is incomplete; details at (:CSSNM00004:) in /u01/app/grid/diag/crs/NODE1/crs/trace/ocssd.trc

2023-07-25 09:16:21.787 [OCSSD(37536)]CRS-7500: The Oracle Grid Infrastructure process 'ocssd' failed to establish Oracle Grid Interprocess Communication (GIPC) high availability connection with remote node 'hyper-ora-ofk2-d-02'.

2023-07-25 09:25:41.510 [OCSSD(37536)]CRS-1609: This node is unable to communicate with other nodes in the cluster and is going down to preserve cluster integrity; details at (:CSSNM00086:) in /u01/app/grid/diag/crs/NODE1/crs/trace/ocssd.trc.
2023-07-25 09:25:41.509 [CSSDAGENT(37501)]CRS-5818: Aborted command 'start' for resource 'ora.cssd'. Details at (:CRSAGF00113:) {0:5:3} in /u01/app/grid/diag/crs/NODE1/crs/trace/ohasd_cssdagent_root.trc.
2023-07-25 09:25:41.563 [OHASD(36613)]CRS-2757: Command 'Start' timed out waiting for response from the resource 'ora.cssd'. Details at (:CRSPE00221:) {0:5:3} in /u01/app/grid/diag/crs/NODE1/crs/trace/ohasd.trc.
2023-07-25 09:25:42.511 [OCSSD(37536)]CRS-1656: The CSS daemon is terminating due to a fatal error; Details at (:CSSSC00012:) in /u01/app/grid/diag/crs/NODE1/crs/trace/ocssd.trc
2023-07-25 09:25:42.512 [OCSSD(37536)]CRS-1603: CSSD on node NODE1 has been shut down.
2023-07-25 09:25:43.299 [OCSSD(37536)]CRS-1609: This node is unable to communicate with other nodes in the cluster and is going down to preserve cluster integrity; details at (:CSSNM00086:) in /u01/app/grid/diag/crs/NODE1/crs/trace/ocssd.trc.

2023-07-25 09:25:43.299 [OCSSD(37536)]CRS-1609: This node is unable to communicate with other nodes in the cluster and is going down to preserve cluster integrity; details at (:CSSNM00086:) in /u01/app/grid/diag/crs/NODE1/crs/trace/ocssd.trc.
2023-07-25T09:25:47.522483+02:00
Errors in file /u01/app/grid/diag/crs/NODE1/crs/trace/ocssd.trc (incident=465):
CRS-8503 [] [] [] [] [] [] [] [] [] [] [] []
Incident details in: /u01/app/grid/diag/crs/NODE1/crs/incident/incdir_465/ocssd_i465.trc

2023-07-25 09:25:47.513 [OCSSD(37536)]CRS-8503: Oracle Clusterware process OCSSD with operating system process ID 37536 experienced fatal signal or exception code 6.

ocssd incident trace
====

----- Invocation Context Dump -----
Address: 0x7fb5c4023e30
Phase: 3
flags: 0x10E0000
Incident ID: 409
Error Descriptor: CRS-8503 [] [] [] [] [] [] [] [] [] [] [] []
Error class: 0
Problem Key # of args: 1
Number of actions: 10
----- Incident Context Dump -----
Address: 0x7fb5c40246d8
Incident ID: 409
Problem Key: CRS 8503
Error: CRS-8503 [] [] [] [] [] [] [] [] [] [] [] []
[00]: dbgexProcessError [diag_dde]
[01]: dbgePostErrorDirectVaList_int [diag_dde]
[02]: dbgePostErrorDirect [diag_dde]
[03]: clsdAdrPostError []
[04]: clsdadrpr_CreateIncidentCheck []
[05]: clsdadrprAlert []
[06]: clsd_malertprintftSig []
[07]: clsbSigErrCB []
[08]: skgesig_sigactionHandler []
[09]: __sighandler []
[10]: gsignal []<-- Signaling
[11]: clssscExit []
[12]: clssscagProcAgReq []
[13]: clssscagAgLsnr []
[14]: clssscthrdmain []
[15]: start_thread []
MD [00]: 'Client ProcId'='ocssd.bin@NODE1.51009_140418811754240' (0x0)
Impact 0:
Impact 1:
Impact 2:
Impact 3:
Derived Impact:
----- END Incident Context Dump -----

Cause

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.