CRS is not coming up as CTSSD is in OFFLINE status (Doc ID 2051836.1)

Last updated on NOVEMBER 11, 2015

Applies to:

Oracle Database - Enterprise Edition - Version 11.2.0.3 and later
Information in this document applies to any platform.
The following entries/logs are for two node cluster, node004, the bad node and node007 where everything is running.

Symptoms

 1.  The "crsctl stat" on node004, the bad node shows that ctssd and rest of the higher stack, such as evmd,crsd,ASM etc are in OFFLINE status :
------------------
# /u01/app/11.2.0.3/grid/bin/crsctl stat res -t -init
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS       
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
      1        ONLINE  OFFLINE                                                   
ora.crf
      1        ONLINE  OFFLINE                                                   
ora.crsd
      1        ONLINE  OFFLINE                                                   
ora.cssd
      1        ONLINE  ONLINE       node004                                     
ora.cssdmonitor
      1        ONLINE  ONLINE       node004                                     
ora.ctssd
      1        ONLINE  OFFLINE                                                   
ora.diskmon
      1        ONLINE  ONLINE       node004                                     
ora.evmd
      1        ONLINE  OFFLINE                                                   
ora.gipcd
      1        ONLINE  ONLINE       node004                                     
ora.gpnpd
      1        ONLINE  ONLINE       node004                                     
ora.mdnsd
      1        ONLINE  ONLINE       node004                                     
root@node004:~#
-----------------


2.  alertnode004.log (node where CRS is not coming up) :
------------
2015-08-31 15:54:22.595:
[cssd(5694)]CRS-1713:CSSD daemon is started in clustered mode
2015-08-31 15:54:27.827:
[ohasd(5349)]CRS-2765:Resource 'ora.crf' has failed on server 'node004'.
.
2015-08-31 15:55:02.343:
[cssd(5694)]CRS-1601:CSSD Reconfiguration complete. Active nodes are node004 node007 .
2015-08-31 15:55:04.934:
[ctssd(6094)]CRS-2407:The new Cluster Time Synchronization Service reference node is host node007.                                          <<=============== Reference or master node of CTSSD
2015-08-31 15:55:04.935:
[ctssd(6094)]CRS-2401:The Cluster Time Synchronization Service started on host node004.
2015-08-31 15:55:08.975:
[ctssd(6094)]CRS-2402:The Cluster Time Synchronization Service aborted on host node004. Details at  in /u01/app/11.2.0.3/grid/log/node004/ctssd/octssd.log.
---------------


3.  NTPD is being used to sync up the clock on the cluster, and its running on the other(good) node, node007(see above) which is the master of CTSSD. So CTSSD is running in OBSERVER mode on node007.
The octssd.log of the bad node, node004  shows that it can't detect the vendor time sync software as "ntp.conf" doesn't exist :
-----------------
2015-08-31 15:55:09.688: [    CTSS][1]Oracle Database CTSS Release 11.2.0.3.0 Production Copyright 2006, 2011 Oracle.  All rights reserved.

2015-08-31 15:55:10.653: [    CTSS][1]sclsctss_gvss8: Return [0] and NTP status [1].
2015-08-31 15:55:10.653: [    CTSS][1]ctss_check_vendor_sw: Vendor time sync software is not detected. status [1].                          <<=============== NTPD or vendor time sync software was not detected.
-----------------

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms