GI/CRS Will Not Start On 2nd Node after a planned reboot (Doc ID 2013790.1)

Last updated on OCTOBER 14, 2016

Applies to:

Oracle Database - Enterprise Edition - Version 11.2.0.3 and later
Information in this document applies to any platform.

Symptoms

After performing a planned reboot the CRS and ASM will not start on second node.

alertdrblade2.log
-----------------------
[ohasd(18392)]CRS-2112:The OLR service started on node drblade2.
2015-05-11 15:48:00.005:
[ohasd(18392)]CRS-1301:Oracle High Availability Service started on node drblade2.
2015-05-11 15:48:00.854:
[ohasd(18392)]CRS-8011:reboot advisory message from host: drblade2, component: cssmonit, with time stamp: L-2015-05-11-11:37:37.483
[ohasd(18392)]CRS-8013:reboot advisory message text: Rebooting after limit 28020 exceeded; disk timeout 27500, network timeout 28020, last heartbeat from CSSD at epoch seconds 1431358629.371, 28111 milliseconds ago based on invariant clock value of 3341653
2015-05-11 15:48:00.885:
[ohasd(18392)]CRS-8017:location: /etc/oracle/lastgasp has 2 reboot advisory log files, 1 were announced and 0 errors occurred
2015-05-11 15:48:02.167:
[/opt/crs/11.2.0.3/grid/bin/oraagent.bin(18432)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/opt/crs/11.2.0.3/grid/log/drblade2/agent/ohasd/oraagent_oracle/oraagent_oracle.log"
2015-05-11 15:53:31.237:
[ohasd(18392)]CRS-5828:Could not start agent '/opt/crs/11.2.0.3/grid/bin/cssdagent_root'. Details at (:CRSAGF00130:) {0:0:2} in /opt/crs/11.2.0.3/grid/log/drblade2/ohasd/ohasd.log.
2015-05-11 15:53:31.249:
[ohasd(18392)]CRS-5828:Could not start agent '/opt/crs/11.2.0.3/grid/bin/cssdmonitor_root'. Details at (:CRSAGF00130:) {0:0:2} in /opt/crs/11.2.0.3/grid/log/drblade2/ohasd/ohasd.log.
2015-05-11 15:53:34.651:
[gpnpd(20617)]CRS-2328:GPNPD started on node drblade2.
2015-05-11 15:53:38.880:
[ohasd(18392)]CRS-2767:Resource state recovery not attempted for 'ora.diskmon' as its target state is OFFLINE
2015-05-11 15:59:05.906:
[ohasd(18392)]CRS-5828:Could not start agent '/opt/crs/11.2.0.3/grid/bin/cssdagent_root'. Details at (:CRSAGF00130:) {0:0:2} in /opt/crs/11.2.0.3/grid/log/drblade2/ohasd/ohasd.log.
2015-05-11 16:04:35.966:
[ohasd(18392)]CRS-5828:Could not start agent '/opt/crs/11.2.0.3/grid/bin/cssdagent_root'. Details at (:CRSAGF00130:) {0:0:2} in /opt/crs/11.2.0.3/grid/log/drblade2/ohasd/ohasd.log.
2015-05-11 16:04:35.968:
[ohasd(18392)]CRS-2758:Resource 'ora.cssd' is in an unknown state.
2015-05-11 16:04:36.001:
[ohasd(18392)]CRS-2807:Resource 'ora.asm' failed to start automatically.
2015-05-11 16:04:36.001:
[ohasd(18392)]CRS-2807:Resource 'ora.cluster_interconnect.haip' failed to start automatically.
2015-05-11 16:04:36.001:
[ohasd(18392)]CRS-2807:Resource 'ora.crsd' failed to start automatically.
2015-05-11 16:04:36.001:
[ohasd(18392)]CRS-2807:Resource 'ora.cssd' failed to start automatically.
2015-05-11 16:04:36.001:
[ohasd(18392)]CRS-2807:Resource 'ora.ctssd' failed to start automatically.
2015-05-11 16:04:36.001:
[ohasd(18392)]CRS-2807:Resource 'ora.evmd' failed to start automatically.


ohasd.log
-------------
2015-05-11 15:53:31.238: [    AGFW][2566100736] {0:0:2} Created alert : (:CRSAGF00130:) :  Failed to start the agent /opt/crs/11.2.0.3/grid/bin/cssdagent_root           
2015-05-11 15:53:31.238: [    AGFW][2566100736] {0:0:2} Agfw Proxy Server sending the last reply to PE for message:RESOURCE_PROBE[ora.cssd 1 1] ID 4097:38
2015-05-11 15:53:31.238: [    AGFW][2566100736] {0:0:2} Can not stop the agent: /opt/crs/11.2.0.3/grid/bin/cssdagent_root because pid is not initialized
2015-05-11 15:53:31.238: [   CRSPE][2555594496] {0:0:2} Received reply to the intial check for: ora.cssd 1 1 on drblade2
2015-05-11 15:53:31.238: [   CRSPE][2555594496] {0:0:2} Initial CHECK has failed for ora.cssd 1 1 on drblade2
2015-05-11 15:53:31.238: [    AGFW][2566100736] {0:0:2} Agfw Proxy Server received the message: RESOURCE_DELETE[ora.cssd 1 1] ID 4358:180
2015-05-11 15:53:31.239: [    AGFW][2566100736] {0:0:2} ora.cssd 1 1 marked as deleted.
2015-05-11 15:53:31.239: [    AGFW][2566100736] {0:0:2} Agfw config version set to: 31
2015-05-11 15:53:31.239: [    AGFW][2566100736] {0:0:2} Could not forward message [RESOURCE_DELETE[ora.cssd 1 1] ID 4358:180] to agent. /opt/crs/11.2.0.3/grid/bin/cssdagent_root is not running
2015-05-11 15:53:31.239: [    AGFW][2566100736] {0:0:2} Deleting the resource: ora.cssd 1 1
2015-05-11 15:53:31.239: [    AGFW][2566100736] {0:0:2} Agfw Proxy Server sending the last reply to PE for message:RESOURCE_DELETE[ora.cssd 1 1] ID 4358:180


Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms