ASM Instance Crashed With Error "terminating the instance due to ORA error 495" (Doc ID 3068746.1)

Last updated on JANUARY 24, 2025

Applies to:

Symptoms

<<<< alert_+ASM2.log

2025-01-11T03:10:46.150331+08:00
NOTE: membership refresh pending for group 1/0x37fa5388 (DATAC1)
2025-01-11T03:10:46.151231+08:00
GMON querying group 1 at 80 for pid 28, osid 162926
2025-01-11T03:10:46.176020+08:00
SUCCESS: refreshed membership for 1/0x37fa5388 (DATAC1)
NOTE: Attempting voting file refresh on diskgroup DATAC1
NOTE: Refresh completed on diskgroup DATAC1. Found 5 voting file(s).
2025-01-11T08:54:16.425923+08:00                                                                   <<<<<<<<<<<<<<<<<<<<<<
NOTE: ASM client XXXXXXX:XXXXX:Cluster-c1 disconnected unexpectedly.      <<<<<<<<<<<<<<<<<<<<<<
NOTE: check client alert log.
..........
Process termination requested for pid 162908 [source = rdbms], [info = 0] [request issued by pid: 162819, uid: 1000]
2025-01-11T08:54:54.259704+08:00
PMON (ospid: 162819): terminating the instance due to ORA error 495 <<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025-01-11T08:54:54.259804+08:00
Cause - 'Instance is being terminated due to fatal process death (pid: 6, ospid: 162855, GEN0)' <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025-01-11T08:54:54.372406+08:00
System state dump requested by (instance=2, osid=162819 (PMON)), summary=[abnormal instance termination]. error - 'Instance is terminating.
'
System State dumped to trace file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_diag_162870.trc <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025-01-11T08:54:55.321784+08:00
Dumping diagnostic data in directory=[cdmp_20250111085454], requested by (instance=2, osid=162819 (PMON)), summary=[abnormal instance termination].
2025-01-11T08:54:55.362771+08:00
License high water mark = 39
2025-01-11T08:54:56.464869+08:00
Instance terminated by PMON, pid = 162819
2025-01-11T08:54:56.465870+08:00
Warning: 2 processes are still attacheded to shmid 6:
(size: 32768 bytes, creator pid: 162007, last attach/detach pid: 162891)
2025-01-11T08:54:57.363478+08:00
USER(prelim) (ospid: 333515): terminating the instance
2025-01-11T08:54:57.363691+08:00
Instance terminated by USER(prelim), pid = 333515
2025-01-11T08:54:59.676660+08:00
* instance_number obtained from CSS = 2, checking for the existence of node 0...
* node 0 does not exist. instance_number = 2
Starting ORACLE instance (normal) (OS id: 333582)
2025-01-11T08:54:59.703136+08:00
****************************************************
Sys-V shared memory will be used for creating SGA
****************************************************
2025-01-11T08:54:59.703539+08:00

<<<< alert_XXXXXX.log

2025-01-11T08:54:10.672194+08:00
Errors in file /u01/app/oracle/diag/rdbms/XXXXXX/XXXXXX/trace/XXXXXX_asmb_271562.trc (incident=481121):
ORA-04031: unable to allocate 3912 bytes of shared memory ("shared pool","unknown object","init_heap_kfsg","ASM extent pointer array")
Incident details in: /u01/app/oracle/diag/rdbms/XXXXXX/XXXXXX/incident/incdir_481121/XXXXXX_asmb_271562_i481121.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
2025-01-11T08:54:11.079279+08:00
NOTE: ASMB0 terminating
2025-01-11T08:54:11.079334+08:00

<<<< XXXXXX_asmb_271562_i481121.trc

HEAP DUMP heap name="init_heap_kfsg" desc=0x525376418
extent sz=0x1040 alt=32767 het=32767 rec=9 flg=0x2 opc=0
parent=0x601057f8 owner=(nil) nex=(nil) xsz=0x4000000 heap=(nil)
fl2=0x24, nex=(nil), idx=6, dsxvers=1, dsxflg=0x0           <<<<<<<<<<<<<
dsx first ext=0x493db4e70
dsx empty ext bytes=0 subheap rc link=0x493db4ee0,0x493db4ee0
pdb id=0, src pdb id=0
Subheap has 1602714016 bytes of memory allocated

....
(latch info) hold_bits=0x4 ud_influx=0x0
Holding:
0x600615c0 'ASM map headers' (level=2, SGA latch)
Holder Location: kffm2.h LINE:363 ID:kffmAllocate:
Holder Slot: slot=2, efd=4, pdb=0
Holder Context: 0
Latch State:
state=busy [value=0x8c]
holder orapid=140, ospid=271562
wlstate=free [value=0]
gotten 7077382 times wait, failed first 19, sleeps 4
gotten 0 times nowait, failed 0, consecutive readers 0
(osp latch info) hold_bits=0x0 ud_influx=0x0
Process Group: DEFAULT, pseudo proc: 0x5f72b5ff8
O/S info: user: oracle, term: UNKNOWN, ospid: 271562
OSD pid info:
.....
Session Wait History:
elapsed time of 0.000000 sec since current wait
0: waited for 'SGA: allocation forcing component growth'
=0x0, =0x0, =0x0
wait_id=21380652 seq_num=16382 snap_id=26
wait times: snap=0.000000 sec, exc=2.502516 sec, total=2.504179 sec
wait times: max=infinite
wait counts: calls=25 os=25
occurred after 0.000000 sec of elapsed time
1: waited for 'SGA: allocation forcing component growth'
=0x0, =0x0, =0x0
wait_id=21380677 seq_num=16381 snap_id=1
wait times: snap=0.000075 sec, exc=0.000075 sec, total=0.000075 sec
wait times: max=infinite
wait counts: calls=1 os=1
.........
----- Incident Context Dump -----
Address: 0x7ffc65c44f88
Incident ID: 481121
Problem Key: ORA 4031
Error: ORA-4031 [3912] [shared pool] [unknown object] [init_heap_kfsg] [ASM extent pointer array] [] [] [] [] [] [] []
[00]: dbgeEndDDEInvocationImpl [diag_dde]
[01]: kghnospc [KGH]<-- Signaling         <<<<<<<<<<<<<
[02]: kghalo [KGH]
[03]: ksmdacnk [KSM]
[04]: ksmdget [KSM]
[05]: kffmXpAlloc [KFF]
[06]: kffmAllocate [KFF]
[07]: kfnbMapOp [KFNU]
[08]: kfnbRun [KFNU]
[09]: ksbrdp [background_proc]
[10]: opirip [OPI]
[11]: opidrv [OPI]
[12]: sou2o []
[13]: opimai_real [OPI]
[14]: ssthrdmain []
[15]: main []
[16]: __libc_start_main []
[17]: 5541F689495641D7 []
.......

==============================================
TOP 10 MEMORY USES FOR SGA HEAP SUB POOL 6
----------------------------------------------
"init_heap_kfsg " 1625 MB 69% <<<<<<<<<<<<<
"ges resource permanent " 383 MB 16%
"free memory " 167 MB 7%
"event statistics per sess " 46 MB 2%
"ASH buffers " 42 MB 2%
"Multiblock Index SO " 24 MB 1%
"dbktb: trace buffer " 22 MB 1%
"gc index split transactio " 19 MB 1%
"value block free list " 5494 KB 0%
"ksdhng: els blkrs cache " 4230 KB 0%
-----------------------------------------
free memory 167 MB
memory alloc. 2201 MB
Sub total 2368 MB
==============================================
TOP 10 MAXIMUM MEMORY USES FOR SGA HEAP SUB POOL 6
----------------------------------------------
"init_heap_kfsg " 1625 MB
"ges resource permanent " 383 MB
"free memory " 236 MB
"KGH: NO ACCESS " 64 MB
"event statistics per sess " 46 MB
"ASH buffers " 42 MB
"Multiblock Index SO " 24 MB
"dbktb: trace buffer " 22 MB
"gc index split transactio " 19 MB
"value block free list " 5494 KB
TOTALS ---------------------------------------
Total free memory 734 MB
Total memory alloc. 5794 MB
Grand total 6528 MB
==============================================

Changes

 N/A

Cause

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.