Root.sh Failed With PROC-26: Error While Accessing the physical Storage Layer (Doc ID 2646396.1)

rootcrs_<NODE>_TIME_STAMP.log
==============================

2020-03-04 21:04:44: Command output:
>
>  ASM has been created and started successfully.
>
>  [DBT-30022] Disk group <ASM_GROUP> mounted successfully.
>
>End Command output
2020-03-04 21:04:44: Configured CRS Home: <GI_HOME>
2020-03-04 21:04:44: Executing cmd: <GI_HOME>/bin/crsctl check resource ora.asm -init
2020-03-04 21:04:45: Executing cmd: <GI_HOME>/bin/crsctl status resource ora.asm -init
2020-03-04 21:04:45: Command output:
>  NAME=ora.asm
>  TYPE=ora.asm.type
>  TARGET=ONLINE
>  STATE=ONLINE on <node>

2020-03-04 21:04:45: Executing the step [ocr_configFirstNode_step_2] to configure OCR on the first node
2020-03-04 21:04:45: Creating or upgrading OCR keys
020-03-05 15:42:45: Executing cmd: <GI_HOME>/bin/clsecho -p has -f clsrsc -m 482 "<GI_HOME>/bin/ocrconfig -upgrade oracle dba"
2020-03-04 21:04:45: Command output:
>  CLSRSC-482: Running command: '<GI_HOME>/bin/ocrconfig -upgrade oracle dba'    >>>>>>>>>>>>>>>>>>>>>>>>>>
>End Command output
2020-03-04 21:04:45: CLSRSC-482: Running command: '<GI_HOME>/bin/ocrconfig -upgrade oracle dba'
2020-03-04 21:04:45: Executing <GI_HOME>/bin/ocrconfig -upgrade oracle dba
2020-03-04 21:04:45: Executing cmd: <GI_HOME>/bin/ocrconfig -upgrade oracle dba
2020-03-04 21:04:25: Command output:
>  PROT-1: Failed to initialize ocrconfig
>  PROC-26: Error while accessing the physical storage Storage layer error [Insufficient quorum to open OCR devices] [0]
>End Command output
2020-03-04 21:04:25: ocrconfig -upgrade failed with status 255
2020-03-04 21:04:25: Executing cmd: <GI_HOME>/bin/clsecho -p has -f clsrsc -m 157
2020-03-04 21:04:26: Command output:
>  CLSRSC-157: Failed to create Oracle Cluster Registry configuration
>End Command output
2020-03-04 21:04:26: CLSRSC-157: Failed to create Oracle Cluster Registry configuration
2020-03-04 21:04:26: Executing cmd: <GI_HOME>/bin/clsecho -p has -f clsrsc -m 259
2020-03-04 21:04:26: Command output:
>  CLSRSC-259: Oracle Clusterware Repository configuration failed
>End Command output
2020-03-04 21:04:26: CLSRSC-259: Oracle Clusterware Repository configuration failed
2020-03-04 21:04:26: ###### Begin DIE Stack Trace ######
2020-03-04 21:04:26:     Package         File                 Line Calling
2020-03-04 21:04:26:     --------------- -------------------- ---- ----------
2020-03-04 21:04:26:  1: main            rootcrs.pl            287 crsutils::dietrap
2020-03-04 21:04:26:  2: oraClusterwareComp::oraocr oraocr.pm             202 main::__ANON__
2020-03-04 21:04:26:  3: oraClusterwareComp oraClusterwareComp.pm   87 oraClusterwareComp::oraocr::configureFirstNode
2020-03-04 21:04:26:  4: crsinstall      crsinstall.pm        2096 oraClusterwareComp::configureCurrentNode
2020-03-04 21:04:26:  5: crsinstall      crsinstall.pm        1963 crsinstall::perform_initial_config
2020-03-04 21:04:26:  6: crsinstall      crsinstall.pm         653 crsinstall::perform_init_config
2020-03-04 21:04:26:  7: crsinstall      crsinstall.pm         813 crsinstall::init_config
2020-03-04 21:04:26:  8: crsinstall      crsinstall.pm         380 crsinstall::CRSInstall
2020-03-04 21:04:26:  9: main            rootcrs.pl            446 crsinstall::new
2020-03-04 21:04:26: ####### End DIE Stack Trace #######
==========
alert.log
==============
2020-03-04 21:04:19.944 [OCSSD(12425)]CRS-1714: Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in <ORACLE_BASE>/diag/crs/<HOST>/crs/trace/ocssd.trc
2020-03-04 21:04:19.178 [ORAROOTAGENT(13206)]CRS-5818: Aborted command 'check' for resource 'ora.storage'. Details at (:CRSAGF00113:) {0:15:2} in <ORACLE_BASE>/diag/crs/<HOST>/crs/trace/ohasd_orarootagent_root.trc.
2020-03-04 21:04:19.058 [OCSSD(12425)]CRS-1714: Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in <ORACLE_BASE>/diag/crs/<HOST>/crs/trace/ocssd.trc
22020-03-04 21:04:19.265 [CSSDAGENT(12417)]CRS-5818: Aborted command 'start' for resource 'ora.cssd'. Details at (:CRSAGF00113:) {0:9:3} in <ORACLE_BASE>/diag/crs/<HOST>/crs/trace/ohasd_cssdagent_root.trc.
2020-03-04 21:04:19.337 [OHASD(11823)]CRS-2757: Command 'Start' timed out waiting for response from the resource 'ora.cssd'. Details at (:CRSPE00221:) {0:9:3} in <ORACLE_BASE>/diag/crs/<HOST>/crs/trace/ohasd.trc.
2020-03-04 21:04:19.267 [OCSSD(12425)]CRS-1656: The CSS daemon is terminating due to a fatal error; Details at (:CSSSC00012:) in <ORACLE_BASE>/diag/crs/<HOST>/crs/trace/ocssd.trc
2020-03-04 21:04:19.268 [OCSSD(12425)]CRS-1652: Starting clean up of CRSD resources.
2020-03-04 21:04:19.275 [OCSSD(12425)]CRS-1653: The clean up of the CRSD resources failed.
2020-03-04 21:04:19.276 [OCSSD(12425)]CRS-8503: Oracle Clusterware process OCSSD with operating system process ID 12425 experienced fatal signal or exception code 6.
2020-03-04 21:04:19.294102+11:00
Errors in file <ORACLE_BASE>/diag/crs/<HOST>/crs/trace/ocssd.trc  (incident=1):
CRS-8503 [] [] [] [] [] [] [] [] [] [] [] []
Incident details in: <ORACLE_BASE>/diag/crs/<HOST>/crs/incident/incdir_1/ocssd_i1.trc
=====================

ocrconfig_<PID>.trc
=====================

cstr=NULL asminst=NULL flags=0x100
2020-03-04 21:04:19.287*:0000030A:     KGF:kgfn.c@5755:kgfnGetBeqDataPriv(): kgfnGetBeqData: found a local instance
2020-03-04 21:04:19.287*:0000030B:     KGF:kgfn.c@5051:kgfnConnect4(): kgfnConnect: srvr valid
2020-03-04 21:04:19.287*:0000030C:     KGF:kgfn.c@5057:kgfnConnect4(): kgfnConnect: bequeath connection
2020-03-04 21:04:19.287*:0000030D:     KGF:kgfn.c@6440:kgfnConnect2Int(): kgfnConnect2Int: sysasm=0 envflags=0x10 srvrflags=0x3 unam=NULL password is NULL pstr=_ocr srvrcallback=0x0 retrynoguid=0
2020-03-04 21:04:19.294*:0000030E:     KGF:kgfn.c@6685:kgfnConnect2Int(): kgfnConnect2Int: cstr=(DESCRIPTION=(ADDRESS=(PROTOCOL=beq)(PROGRAM=<GI_HOME>/bin/oracle)(ARGV0=oracle+ASM1_ocr)(ENVS='ORACLE_HOME=<GI_HOME>,ORACLE_SID=+ASM1')(ARGS='(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))')(PRIVS=(USER=oracle)(GROUP=dba)))(enable=setuser))
2020-03-04 21:04:19.304*:0000030F:     KGF:kgfn.c@6698:kgfnConnect2Int(): kgfnConnect2Int: OCIServerAttach failed
2020-03-04 21:04:19.304*:00000310:     KGF:kgfn.c@6731:kgfnConnect2Int(): kgfnConnect2Int: OCIServerAttach return=-1
2020-03-04 21:04:19.304*:00000311:     KGF:kgfn.c@1660:kgfnRecordErrPriv(): kgfnRecordErrPriv: status=-1  at kgfn.c:6980
2020-03-04 21:04:20.313*:00000312:     KGF:kgfn.c@1706:kgfnRecordErrPriv(): kgfnRecordErrPriv: 12560 error=ORA-12560: TNS:protocol adapter error

2020-03-04 21:04:20.313*:00000313:     KGF:kgfn.c@1742:kgfnRecordErrPriv(): kgfnRecordErrPriv: rec=1
2020-03-04 21:04:20.313*:00000314:     KGF:kgfn.c@5253:kgfnConnect4(): kgfnConnect2: failed to connect
2020-03-04 21:04:20.313*:00000315:     KGF:kgfn.c@5272:kgfnConnect4(): kgfnConnect: conn=0x0
2020-03-04 21:04:20.314*:00000316:     KGF:kgfp.c@729:kgfpInitComplete3(): kgfpInitComplete2 hdl=0x1007317d8 conn=0x1007317f0 ok=0
2020-03-04 21:04:20.314*:00000317:     KGF:kgfo.c@989:kgfo_kge2slos2(): kgfo_kge2slos error stack at kgfoAl06: ORA-12560: TNS:protocol adapter error
ORA-12560: TNS:protocol adapter error
ORA-27300: OS system dependent operation:sslssunreghdlr failed with status: 0
ORA-27301: OS failure message: Error 0
ORA-27302: failure occurred at: sskgpreset1

2020-03-04 21:04:20.314*:00000318:     KGF:kgfo.c@1100:kgfoSaveErrorInt(): kgfoSaveError: ctx=0x100670060 hdl=0x0 gph=0x1006fa698 ose=0xffffffff7ffe84bc at kgfo.c:1048
2020-03-04 21:04:20.314*:00000319:     KGF:kgfo.c@1157:kgfoSaveErrorInt(): kgfoSaveError: ignoring existing error:
ORA-12560: TNS:protocol adapter error
ORA-12560: TNS:protocol adapter error
ORA-27300: OS system dependent operation:sslssunreghdlr failed with status: 0
ORA-27301: OS failure message: Error 0
ORA-27302: failure occurred at: sskgpreset1
=====================