My Oracle Support Banner

12.2 Oracle Restart: CRS-8503 and OCSSD Experienced Fatal Signal or Exception Code 11 and Abort Instance with ORA-15064 and ORA-29710 (Doc ID 2403736.1)

Last updated on DECEMBER 20, 2022

Applies to:

Oracle Database Cloud Schema Service - Version N/A and later
Oracle Database Exadata Cloud Machine - Version N/A and later
Oracle Database Exadata Express Cloud Service - Version N/A and later
Oracle Cloud Infrastructure - Database Service - Version N/A and later
Oracle Database Cloud Exadata Service - Version N/A and later
Information in this document applies to any platform.

Symptoms

Instance abort with ORA-15064: communication failure with ASM instance. 

alert_ORCL.log inside <oracle base>/diag/rdbms/orcl/ORCL/trace location:

2018-05-16T20:00:25.315818-03:00
NOTE: ASMB0 terminating
2018-05-16T20:00:25.315902-03:00
Errors in file <oracle base>/rdbms/orcl/ORCL/trace/ORCL_asmb_5479.trc:
ORA-15064: communication failure with ASM instance
ORA-03113: end-of-file on communication channel
.
USER (ospid: 5479): terminating the instance due to error 15064

ASM alert log having ORA-29710 logged just prior to the above incident time 

<oracle base>/diag/asm/+asm/+ASM/trace/alert_+ASM.log:

2018-05-16T20:00:24.643180-03:00
Errors in file <oracle base>/diag/asm/+asm/+ASM/trace/+ASM_gmon_5174.trc:
ORA-29710: Current operation aborted by Cluster Synchronization Services
Errors in file <oracle base>/diag/asm/+asm/+ASM/trace/+ASM_gmon_5174.trc (incident=48161):
ORA-29710 [] [] [] [] [] [] [] [] [] [] [] []
Incident details in: <oracle base>/diag/asm/+asm/+ASM/incident/incdir_48161/+ASM_gmon_5174_i48161.trc
2018-05-16T20:00:25.301868-03:00
USER (ospid: 5174): terminating the instance due to error 29710
2018-05-16T20:00:25.314285-03:00
System state dump requested by (instance=1, osid=5174 (GMON)), summary=[abnormal instance termination].
System State dumped to trace file <oracle base>/diag/asm/+asm/+ASM/trace/+ASM_diag_5148_20180516200025.trc

Prior to above incident time, Clusterware alert log and cssd incident trace indicate below messages 

<oracle base>/diag/crs/<host name>/crs/trace/alert.log:

2018-05-16 20:00:22.407 [ORAAGENT(3344)]CRS-5822: Agent '$GRID_HOME/bin/oraagent_grid' disconnected from server. Details at (:CRSAGF00117:) {0:7:8} in <oracle base>/diag/crs/nodea/crs/trace/ohasd_oraagent_grid.trc.
2018-05-16 20:00:22.407 [CSSDAGENT(3866)]CRS-5822: Agent '$GRID_HOME/bin/cssdagent_grid' disconnected from server. Details at (:CRSAGF00117:) {0:9:873} in <oracle base>/diag/crs/nodea/crs/trace/ohasd_cssdagent_grid.trc.
2018-05-16 20:00:23.280 [OCSSD(3900)]CRS-1656: The CSS daemon is terminating due to a fatal error; Details at (:CSSSC00012:) in <oracle base>/diag/crs/nodea/crs/trace/ocssd.trc
2018-05-16 20:00:23.281 [OCSSD(3900)]CRS-8503: Oracle Clusterware process OCSSD with operating system process ID 3900 experienced fatal signal or exception code 11.
2018-05-16T20:00:23.297260-03:00
Errors in file <oracle base>/diag/crs/nodea/crs/trace/ocssd.trc (incident=201):
CRS-8503 [clssnmCheckVFStatus()+46 clssnm1.c:4181] [Signal/Exception: 11] [Instruction Addr: 0x420cb6] [Memory Addr: (nil)] [] [] [] [] [] [] [] []
Incident details in: <oracle base>/diag/crs/nodea/crs/incident/incdir_201/ocssd_i201.trc

<oracle base>/diag/crs/<host name>/crs/incident/incdir_201/ocssd_i201.trc:

Jump to table of contents
Dump continued from file: <oracle base>/diag/crs/nodea/crs/trace/ocssd.trc
[TOC00001]
CRS-8503 [clssnmCheckVFStatus()+46 clssnm1.c:4181] [Signal/Exception: 11] [Instruction Addr: 0x420cb6] [Memory Addr: (nil)] [] [] [] [] [] [] [] []
[TOC00001-END]

----- START DDE Action: 'clsdAdrActions' (Sync) -----
default:3268093696: clsssc_sighandler_common: CSSD handling internal error CRS-8503 (signal 11)

2018-05-16 20:00:23.323 :CSSD:3268093696:

----- Call Stack Trace -----
2018-05-16 20:00:23.323 :CSSD:3268093696: calling call entry argument values in hex
2018-05-16 20:00:23.323 :CSSD:3268093696: location type point (? means dubious value)
2018-05-16 20:00:23.323 :CSSD:3268093696: -------------------- -------- -------------------- ----------------------------
...
2018-05-16 20:00:23.340 :CSSD:3268093696: __sighandler() call skgesig_sigactionHa 000000400 000000000 000000000
2018-05-16 20:00:23.340 :CSSD:3268093696: ndler() 7FB6C2C88720 ? 001FC5650 ?
2018-05-16 20:00:23.340 :CSSD:3268093696: 001FC5610 ?
2018-05-16 20:00:23.340 :CSSD:3268093696: clssnmCheckVFStatus signal __sighandler() 002722410 000000000 000000000
2018-05-16 20:00:23.340 :CSSD:3268093696: ()+46 clssnm1.c:418 000000000 ? 7FB6C2C88F32 ?
2018-05-16 20:00:23.340 :CSSD:3268093696: 1 000000001 ?
2018-05-16 20:00:23.340 :CSSD:3268093696: clssnmDiagDump()+20 call clssnmCheckVFStatus 002722410 000000000
2018-05-16 20:00:23.340 :CSSD:3268093696: 2 clssnm.c:5231 () 000000000 ? 000000000 ?
2018-05-16 20:00:23.340 :CSSD:3268093696: 7FB6C2C88F32 ? 000000001 ? 

 

Changes

 

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!

In this Document
Symptoms
Changes
Cause
Solution
 Community Discussions
References


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.