My Oracle Support Banner

10g/11gR1: Healthy Instances May Die With Error ORA-29702 When Other Instances Are Hung (Doc ID 789196.1)

Last updated on FEBRUARY 03, 2019

Applies to:

Oracle Server - Enterprise Edition - Version to [Release 10.1 to 11.1]
Information in this document applies to any platform.


RAC Instances (ASM and RDBMS) may crash on hang with a reconfiguration timeout; for example, some instances get hung due to OS or other issues during the reconfiguration, which prevents the reconfiguration phase from completing.

The alert log of crashed RAC instance will show :

Thu Feb 12 19:47:12 2009
Reconfiguration started (old inc 5, new inc 6)
List of nodes:
0 2 3 4
Global Resource Directory frozen
* dead instance detected - domain 1 invalid = TRUE
* dead instance detected - domain 2 invalid = TRUE
* dead instance detected - domain 3 invalid = TRUE
Communication channels reestablished
Thu Feb 12 20:03:56 2009
NOTE: database cdfm1p3:cdfm1p failed during msg 19, reply 2
Thu Feb 12 20:04:52 2009
Error: KGXGN polling error (15)
Thu Feb 12 20:04:52 2009
Errors in file /opt/oracle/admin/+ASM/bdump/+asm3_lmon_4479.trc:
ORA-29702: error occurred in Cluster Group Service operation
LMON: terminating instance due to error 29702
Thu Feb 12 20:04:56 2009
Dump system state for local instance only
System State dumped to trace file /opt/oracle/admin/+ASM/bdump/+asm3_diag_4475.
Thu Feb 12 20:04:57 2009
Trace dumping is performing id=[cdmp_20090212200457]
Thu Feb 12 20:04:58 2009
Instance terminated by LMON, pid = 4479

The LMON tracefile of the crashed instances will show :

kjxgmpoll: terminate the CGS reconfig.
Error: KGXGN polling error (15)
error 29702 detected in background process
ORA-29702: error occurred in Cluster Group Service operation
ksuitm: waiting up to [5] seconds before killing DIAG

Note the ocssd.log may not contain any obvious information for instance crash.


To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!

In this Document

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.