ora.crsd is not coming up due to wrong permission of /var (Doc ID 2124248.1)

Last updated on APRIL 06, 2016

Applies to:

Oracle Database - Enterprise Edition - Version 11.2.0.4 and later
Information in this document applies to any platform.

Symptoms

After restart, CRSD is not coming up:

$ crsctl stat res -t -init

Cluster Resources
--------------------------------------------------------------------------------
ora.asm
1 ONLINE ONLINE racnode3 Started
ora.cluster_interconnect.haip
1 ONLINE ONLINE racnode3
ora.crf
1 ONLINE ONLINE racnode3
ora.crsd
1 ONLINE OFFLINE
ora.cssd
1 ONLINE ONLINE racnode3
....

 

crsd.log

2016-03-05 18:07:53.535: [ CRSMAIN][3913283360] Checking the OCR device
2016-03-05 18:07:53.535: [ CRSMAIN][3913283360] Sync-up with OCR
2016-03-05 18:07:53.535: [ CRSMAIN][3913283360] Connecting to the CSS Daemon
2016-03-05 18:07:53.535: [ CRSMAIN][3913283360] Getting local node number
2016-03-05 18:07:53.535: [ CRSMAIN][3906832128] Policy Engine is not initialized yet!
2016-03-05 18:07:53.536: [ CRSMAIN][3913283360] Initializing OCR
[ CLWAL][3913283360]clsw_Initialize: OLR initlevel [70000]
2016-03-05 18:07:53.903: [ OCRASM][3913283360]proprasmo: Error in open/create file in dg [OCRVD]
[ OCRASM][3913283360]SLOS : SLOS: cat=8, opn=kgfoOpen01, dep=29701, loc=kgfokge ====>>>> 29701 - 29701, 00000, "unable to connect to Cluster Synchronization Service"

2016-03-05 18:07:53.903: [ OCRASM][3913283360]ASM Error Stack :
2016-03-05 18:07:53.945: [ OCRRAW][3913283360]proprioo: Failed to open [+OCRVD]. Returned proprasmo() with [26]. Marking location as UNAVAILABLE.
2016-03-05 18:07:53.945: [ OCRRAW][3913283360]proprioo: No OCR/OLR devices are usable
2016-03-05 18:07:53.945: [ OCRASM][3913283360]proprasmcl: asmhandle is NULL
2016-03-05 18:07:53.946: [ GIPC][3913283360] gipcCheckInitialization: possible incompatible non-threaded init from [prom.c : 694], original from [clsss.c : 5358]
2016-03-05 18:07:53.947: [ default][3913283360]clsvactversion:4: Retrieving Active Version from local storage.
2016-03-05 18:07:53.950: [ OCRRAW][3913283360]proprrepauto: The local OCR configuration matches with the configuration published by OCR Cache Writer. No repair required.
2016-03-05 18:07:53.951: [ OCRRAW][3913283360]proprinit: Could not open raw device
2016-03-05 18:07:53.951: [ OCRASM][3913283360]proprasmcl: asmhandle is NULL
2016-03-05 18:07:53.954: [ OCRAPI][3913283360]a_init:16!: Backend init unsuccessful : [26]
2016-03-05 18:07:53.954: [ CRSOCR][3913283360] OCR context init failure. Error: PROC-26: Error while accessing the physical storage

2016-03-05 18:07:53.955: [ CRSD][3913283360] Created alert : (:CRSD00111:) : Could not init OCR, error: PROC-26: Error while accessing the physical storage

2016-03-05 18:07:53.955: [ CRSD][3913283360][PANIC] CRSD exiting: Could not init OCR, code: 26
2016-03-05 18:07:53.955: [ CRSD][3913283360] Done.

 

ocrcheck and ocrdump is working fine:

Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 4336
Available space (kbytes) : 257784
ID : 1107908941
Device/File Name : +OCRVD
Device/File integrity check succeeded

Device/File not configured

Device/File not configured

Device/File not configured

Device/File not configured

Cluster registry integrity check succeeded

Logical corruption check succeeded

 

CRSD is up on all other nodes:

Cluster Resources
--------------------------------------------------------------------------------
ora.asm
1 ONLINE ONLINE racnode2 Started
....
ora.crsd
1 ONLINE ONLINE racnode2
ora.cssd
1 ONLINE ONLINE racnode2

  

ASM started fine and mounted all diskgroup:

Sat Mar 05 18:07:43 2016
MMNL started with pid=21, OS id=26597
lmon registered with NM - instance number 5 (internal mem no 4)
Reconfiguration started (old inc 0, new inc 80)
ASM instance
List of instances:
1 2 3 4 5 6 (myinst: 5)
Global Resource Directory frozen
* allocate domain 0, invalid = TRUE
....
Reconfiguration complete
Sat Mar 05 18:07:44 2016
....
NOTE: Diskgroup used for Voting files is:
OCRVD
Diskgroup with spfile:OCRVD
Diskgroup used for OCR is:OCRVD
....
NOTE: cache mounting group 3/0xB42865C6 (OCRVD) succeeded
NOTE: cache ending mount (success) of group OCRVD number=3 incarn=0xb42865c6

 

+ASM3_ora_<pid>.trc 

2016-03-21 13:39:13.319: [ GIPCNET] gipcmodNetworkProcessConnect: [network] failed connect attempt endp 0xc1f0a70 [0000000000000018] { gipcEndpoint : localAddr 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=)(GIPCID=00000000-00000000-0))', remoteAddr 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_racnode3_)(GIPCID=00000000-00000000-0))', numPend 0, numReady 1, numDone 0, numDead 0, numTransfer 0, objFlags 0x0, pidPeer 0, readyRef (nil), ready 0, wobj 0xc1f1ea0, sendp 0xc1f1ce0flags 0x802c8712, usrFlags 0x34000 }, req 0xc1f24c0 [000000000000001d] { gipcConnectRequest : addr 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_racnode3_)(GIPCID=00000000-00000000-0))', parentEndp 0xc1f0a70, ret gipcretFail (1), objFlags 0x0, reqFlags 0x2 }
2016-03-21 13:39:13.321: [ GIPCNET] gipcmodNetworkProcessConnect: slos op : sgipcnDSConnectHelper
2016-03-21 13:39:13.321: [ GIPCNET] gipcmodNetworkProcessConnect: slos dep : Permission denied (13)
2016-03-21 13:39:13.321: [ GIPCNET] gipcmodNetworkProcessConnect: slos loc : connect
2016-03-21 13:39:13.321: [ GIPCNET] gipcmodNetworkProcessConnect: slos info: failed to /var/tmp/.oracle/sOCSSD_LL_racnode3_
2016-03-21 13:39:13.322: [ CSSCLNT]clssscConnect: gipc request failed with 1 (0x12)
2016-03-21 13:39:13.323: [ CSSCLNT]clsssInitNative: connect to (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_racnode3_)) failed, rc 1
kgxgncin: CLSS init failed with status 3
kgxgncin: return status 3 (1311719766 SKGXN not av) from CLSS
NOTE: kfmsInit: ASM failed to initialize group services

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms