ASM crash instead of force dismount affected diskgroup on metadata corruption (Doc ID 2169448.1)

Last updated on AUGUST 19, 2016

Applies to:

Oracle Database - Enterprise Edition - Version 11.2.0.3 to 12.1.0.1 [Release 11.2 to 12.1]
Information in this document applies to any platform.

Symptoms

ASM instance failed to start when there is a metadata corruption on one of the diskgroup.

An extract from ASM alert log showing the list of diskgroups during startup


SQL> ALTER DISKGROUP ALL MOUNT /* asm agent call crs *//* {0:9:489} */
NOTE: Diskgroups listed in ASM_DISKGROUPS are
DATA
FRA
RECO
ORCLTST
RACONE
OLTPPRD
RACUAT
RACTST
DEVTST
RACPRD
NOTE: Diskgroup used for Voting files is:
OCR_VOTE
Diskgroup with spfile:OCR_VOTE

 

Amoung the above diskgroups, diskgroup 'RACUAT' having problems and ASM instance failed to start with ORA-00600 errors and finally RBAL terminating the instance due to error 488. 

Mon Aug 01 16:10:51 2016
Dumping diagnostic data in directory=[cdmp_20160801161051], requested by (instance=1, osid=27227 (RBAL)), summary=[incident=659997].
ERROR: check of diskgroup RACUAT found 1 total errors
ORA-15049: diskgroup "RACUAT" contains 1 error(s)
Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_rbal_27227.trc (incident=659998):
ORA-00600: internal error code, arguments: [17090], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/asm/+asm/+ASM1/incident/incdir_659998/+ASM1_rbal_27227_i659998.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
ERROR: ORA-600 in COD recovery for diskgroup 22/0x874bed4a (RACUAT)
ERROR: ORA-600 thrown in RBAL for group number 22
Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_rbal_27227.trc:
ORA-00600: internal error code, arguments: [17090], [], [], [], [], [], [], [], [], [], [], []
Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_rbal_27227.trc:
ORA-00600: internal error code, arguments: [17090], [], [], [], [], [], [], [], [], [], [], []
RBAL (ospid: 27227): terminating the instance due to error 488
System state dump requested by (instance=1, osid=27227 (RBAL)), summary=[abnormal instance termination].
System State dumped to trace file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_diag_27165_20160801161055.trc
Dumping diagnostic data in directory=[cdmp_20160801161055], requested by (instance=1, osid=27227 (RBAL)), summary=[abnormal instance termination].
Instance terminated by RBAL, pid = 27227
Mon Aug 01 16:11:06 2016
* instance_number obtained from CSS = 1, checking for the existence of node 0...
* node 0 does not exist. instance_number = 1
Starting ORACLE instance (normal)

 

Extract from RBAL trace 'ASM1_rbal_27227.trc'

*** 2016-08-01 16:10:20.626
Incident 659997 created, dump file: /u01/app/oracle/diag/asm/+asm/+ASM1/incident/incdir_659997/+ASM1_rbal_27227_i659997.trc
ORA-00600: internal error code, arguments: [kfdAuDealloc2], [189], [5102], [0], [RACUAT], [], [], [], [], [], [], []

*** 2016-08-01 16:10:54.410
Incident 659998 created, dump file: /u01/app/oracle/diag/asm/+asm/+ASM1/incident/incdir_659998/+ASM1_rbal_27227_i659998.trc
ORA-00600: internal error code, arguments: [17090], [], [], [], [], [], [], [], [], [], [], []

ORA-00600: internal error code, arguments: [17090], [], [], [], [], [], [], [], [], [], [], []
error 488 detected in background process
ORA-00600: internal error code, arguments: [17090], [], [], [], [], [], [], [], [], [], [], []
kjzduptcctx: Notifying DIAG for crash event

----- Abridged Call Stack Trace -----

ksedsts <- kjzdssdmp <- kjzduptcctx <- kjzdicrshnfy <- ksuitm
<- ksbrdp <- opirip <- opidrv <- sou2o <- opimai_real
<- ssthrdmain <- main

*** 2016-08-01 16:10:55.282
RBAL (ospid: 27227): terminating the instance due to error 488
ksuitm: waiting up to [5] seconds before killing DIAG(27165)

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms