12.1.0.2 Grid Infrastructure Stack not Start due to .pid file issues (Doc ID 2028511.1)

Last updated on OCTOBER 14, 2016

Applies to:

Oracle Database - Enterprise Edition - Version 12.1.0.2 and later
Information in this document applies to any platform.

Symptoms

On a 12.1.0.2 multi node cluster, when starting the clusterware, GI stack fails to start, this could happen to any of the resource managed by OHASD eg: ora.mdnsd (mdnsd.bin), ora.gipcd (gipcd.bin), ora.gpnpd (gpnpd.bin), ora.evmd (evmd.bin) etc.

Case I. No messages is written to the corresponding  <process>.trc file. The node alert log (<GI home>/log/<node>/alert<node>.log) shows the following messages:

2015-07-07 14:43:22.594 [ORAAGENT(4642)]CRS-5818: Aborted command 'start' for resource 'ora.gipcd'. Details at (:CRSAGF00113:) {0:0:2} in /u01/app/grid/diag/crs/racnode1/crs/trace/ohasd_oraagent_grid.trc.
2015-07-07 14:43:23.496 [ORAAGENT(4642)]CRS-5017: The resource action "ora.gipcd start" encountered the following error:
2015-07-07 14:43:23.496+Start action for daemon aborted. For details refer to "(:CLSN00107:)" in "/u01/app/grid/diag/crs/racnode1/crs/trace/ohasd_oraagent_grid.trc".
2015-07-07 14:43:26.803 [OHASD(4253)]CRS-2757: Command 'Start' timed out waiting for response from the resource 'ora.gipcd'. Details at (:CRSPE00163:) {0:0:2} in /u01/app/grid/diag/crs/racnode1/crs/trace/ohasd.trc.

 

Case II. <ORACLE_BASE>/diag/crs/<host>/crs/trace/gipcd.trc shows:

2015-10-26 11:58:05.901092 : CLSDMT:180586240: PID for the Process [5395], connkey 13
2015-10-26 11:58:05.901470 : CLSDMT:180586240: Failed to record pid for GIPCD
2015-10-26 11:58:05.901474 : CLSDMT:180586240: Terminating process
...
trace file /home/grid/oraclebase/diag/crs/racnode1/crs/trace/gipcd.trc
Oracle Database 12c Clusterware Release 12.1.0.2.0 - Production Copyright 1996, 2014 Oracle. All rights reserved.
DDE: Flood control is not active
2015-10-26 11:59:05.909217 : GIPCD:180586240: gipcd_ExitCB: one or more of gipcdThreads failed to come into offline in 60 seconds of time, aborting the gipcd process
CLSB:180586240: Oracle Clusterware infrastructure error in GIPCD (OS PID 5395): Fatal signal 6 has occurred in program gipcd thread 180586240; nested signal count is 1
Incident 33 created, dump file: /home/grid/oraclebase/diag/crs/racnode1/crs/incident/incdir_33/gipcd_i33.trc
CRS-8503 [] [] [] [] [] [] [] [] [] [] [] []


Case III.  crsctl start crs shows:

Start action for daemon aborted. For details refer to "(:CLSN00107:)" in "D:\app\grid\racdbauser\diag\crs\racnode1\crs\trace\ohasd_oraagent_system.trc".
CRS-2674: Start of 'ora.mdnsd' on 'racnode1' failed
CRS-2679: Attempting to clean 'ora.mdnsd' on 'racnode1'
CRS-2681: Clean of 'ora.mdnsd' on 'racnode1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'racnode1'
CRS-5017: The resource action "ora.gpnpd start" encountered the following error:

Start action for daemon aborted. For details refer to "(:CLSN00107:)" in "D:\app\grid\racdbauser\diag\crs\racnode1\crs\trace\ohasd_oraagent_system.trc".
CRS-2883: Resource 'ora.gpnpd' failed during Clusterware stack start.
CRS-4406: Oracle High Availability Services synchronous start failed.
CRS-4000: Command Start failed, or completed with errors.
2015/11/30 18:32:21 CLSRSC-117: Failed to start Oracle Clusterware stack

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms