10g/11gR1: Healthy Instances May Die With Error ORA-29702 When Other Instances Are Hung (Doc ID 789196.1)

Last updated on JULY 12, 2012

Applies to:

Oracle Server - Enterprise Edition - Version 10.1.0.2 to 11.1.0.6 [Release 10.1 to 11.1]
Information in this document applies to any platform.

Symptoms

RAC Instances (ASM and RDBMS) may crash on hang with a reconfiguration timeout; for example, some instances get hung due to OS or other issues during the reconfiguration, which prevents the reconfiguration phase from completing.

The alert log of crashed RAC instance will show :

Thu Feb 12 19:47:12 2009
Reconfiguration started (old inc 5, new inc 6)
List of nodes:
0 2 3 4
Global Resource Directory frozen
* dead instance detected - domain 1 invalid = TRUE
* dead instance detected - domain 2 invalid = TRUE
* dead instance detected - domain 3 invalid = TRUE
Communication channels reestablished
Thu Feb 12 20:03:56 2009
NOTE: database cdfm1p3:cdfm1p failed during msg 19, reply 2
Thu Feb 12 20:04:52 2009
Error: KGXGN polling error (15)
Thu Feb 12 20:04:52 2009
Errors in file /opt/oracle/admin/+ASM/bdump/+asm3_lmon_4479.trc:
ORA-29702: error occurred in Cluster Group Service operation
LMON: terminating instance due to error 29702
Thu Feb 12 20:04:56 2009
Dump system state for local instance only
System State dumped to trace file /opt/oracle/admin/+ASM/bdump/+asm3_diag_4475.
trc
Thu Feb 12 20:04:57 2009
Trace dumping is performing id=[cdmp_20090212200457]
Thu Feb 12 20:04:58 2009
Instance terminated by LMON, pid = 4479


The LMON tracefile of the crashed instances will show :

kjxgmpoll: terminate the CGS reconfig.
Error: KGXGN polling error (15)
error 29702 detected in background process
ORA-29702: error occurred in Cluster Group Service operation
ksuitm: waiting up to [5] seconds before killing DIAG


Note the ocssd.log may not contain any obvious information for instance crash.



Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms