After Private Network is Restored, CSS Started But "CRS-2878:Failed to restart resource 'ora.cssd'" Causing CRSD to go Offline (Doc ID 1921160.1)

Last updated on SEPTEMBER 03, 2014

Applies to:

Oracle Database - Enterprise Edition - Version 11.2.0.4 and later
Information in this document applies to any platform.

Symptoms

11gR2 Grid Infrastructure node 2 evicted as expected after private network issue, however, after network is restored,
ora.cssd and ora.crsd on the evicted node fails to start.

• <GI_HOME>/log/<node>/alert<node>.log - evicted node

2014-07-25 11:12:26.510:
[cssd(12284)]CRS-1713:CSSD daemon is started in clustered mode
...
2014-07-25 11:15:44.348:
[cssd(12284)]CRS-1601:CSSD Reconfiguration complete. Active nodes are wvfyapp1 wvfyapp2 .
2014-07-25 11:15:45.432:
[ohasd(22341)]CRS-2878:Failed to restart resource 'ora.cssd'  ######## CSSD started and reconfig complete, but reported CSSD failed to start immediately.
2014-07-25 11:15:47.523:
[ctssd(13009)]CRS-2403:The Cluster Time Synchronization Service on host wvfyapp2 is in observer mode.
2014-07-25 11:15:47.927:
[ctssd(13009)]CRS-2407:The new Cluster Time Synchronization Service reference node is host wvfyapp1.
2014-07-25 11:15:47.929:
[ctssd(13009)]CRS-2401:The Cluster Time Synchronization Service started on host wvfyapp2.
2014-07-25 11:15:48.596:
[/oracle/app/11.2.0/grid/bin/oraagent.bin(22560)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/oracle/app/11.2.0/grid/log/wvfyapp2/agent/ohasd/oraagent_grid/oraagent_grid.log"
2014-07-25 11:15:49.512:
[ohasd(22341)]CRS-2767:Resource state recovery not attempted for 'ora.diskmon' as its target state is OFFLINE
2014-07-25 11:15:49.512:
[ohasd(22341)]CRS-2769:Unable to failover resource 'ora.diskmon'.
2014-07-25 11:15:54.417:
[/oracle/app/11.2.0/grid/bin/oraagent.bin(22560)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/oracle/app/11.2.0/grid/log/wvfyapp2/agent/ohasd/oraagent_grid/oraagent_grid.log"
2014-07-25 11:15:59.046:
[ctssd(13009)]CRS-2409:The clock on host wvfyapp2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.
2014-07-25 11:16:10.714:
[ohasd(22341)]CRS-2878:Failed to restart resource 'ora.crsd'

• <GI_HOME>/log/<node>/crsd/crsd.log - evicted node

2014-07-25 11:16:09.940: [ CRSMAIN][1] First attempt: init CSS context succeeded.
[ clsdmt][3]Listening to (ADDRESS=(PROTOCOL=ipc)(KEY=wvfyapp2DBG_CRSD))
2014-07-25 11:16:09.949: [ clsdmt][3]PID for the Process [13147], connkey 1
2014-07-25 11:16:09.950: [ clsdmt][3]Creating PID [13147] file for home /oracle/app/11.2.0/grid host wvfyapp2 bin crs to /oracle/app/11.2.0/grid/crs/init/
2014-07-25 11:16:09.950: [ clsdmt][3]Writing PID [13147] to the file [/oracle/app/11.2.0/grid/crs/init/wvfyapp2.pid]
2014-07-25 11:16:10.681: [ CRSMAIN][3] Policy Engine is not initialized yet!
2014-07-25 11:16:10.681: [ CRSMAIN][1] CRS Daemon Starting
2014-07-25 11:16:10.682: [ CRSD][1] Logging level for Module: allcomp 0
2014-07-25 11:16:10.682: [ CRSD][1] Logging level for Module: default 0

...

2014-07-25 11:16:10.685: [ CRSD][1] Logging level for Module: OCRASM 1
2014-07-25 11:16:10.685: [ CRSMAIN][1] Checking the OCR device
2014-07-25 11:16:10.686: [ CRSMAIN][1] Sync-up with OCR
2014-07-25 11:16:10.686: [ CRSMAIN][1] Connecting to the CSS Daemon
2014-07-25 11:16:10.686: [ CRSMAIN][1] Getting local node number
2014-07-25 11:16:10.686: [ CRSMAIN][3] Policy Engine is not initialized yet!
2014-07-25 11:16:10.687: [ CRSMAIN][1] Initializing OCR
[ CLWAL][1]clsw_Initialize: OLR initlevel [70000]
2014-07-25 11:16:10.765: [ CRSD][3] CRSD exiting on stop request from default

######## CRSD fails to start.

2014-07-25 11:16:10.765: [ CRSD][3] Done.

 • <GI_HOME>/log/<node>/ohasd/ohasd.log - evicted node

2014-07-25 11:15:45.364: [ CRSPE][28]{0:9:16} CRS-2676: Start of 'ora.cssd' on 'wvfyapp2' succeeded

2014-07-25 11:15:45.432: [ CRSPE][28]{0:9:15} Re-evaluation of queued op [START of [ora.cssd 1 1] on [wvfyapp2] : local=1, unplanned=06000000002403830]. found it no longer needed:CRS-2506: Operation on 'START of [ora.cssd 1 1] on [wvfyapp2] : local=1, unplanned=06000000002403830' has been cancelled
. Finishing the op.
2014-07-25 11:15:45.432: [ INIT][28]{0:9:15} {0:9:15} Created alert : (:CRSPE00190:) : Resource restart failed!

######## CSSD start succeeded, but reported error CRS-2506 immediately and CSSD failed.

...

2014-07-25 11:16:10.711: [ CRSPE][28]{0:9:16} CRS-2676: Start of 'ora.crsd' on 'wvfyapp2' succeeded

2014-07-25 11:16:10.714: [ CRSPE][28]{0:35:12} Re-evaluation of queued op [START of [ora.crsd 1 1] on [wvfyapp2] : local=1, unplanned=060000000024be760]. found it no longer needed:CRS-2506: Operation on 'START of [ora.crsd 1 1] on [wvfyapp2] : local=1, unplanned=060000000024be760' has been cancelled
. Finishing the op.
2014-07-25 11:16:10.714: [ INIT][28]{0:35:12} {0:35:12} Created alert : (:CRSPE00190:) : Resource restart failed!

######## CRSD start succeeded, but reported error CRS-2506 immediately and CRSD failed.

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms