12.2: DB Instance crashed with 'SGA: allocation forcing component growth' (Doc ID 2316296.1)

2017-09-30T10:33:40.515296+02:00
LMD1 (ospid: <PID>) waits for event 'SGA: allocation forcing component growth' for 86 secs.
2017-09-30T10:33:45.868191+02:00
Errors in file $TRACE/<SID>_lmhb_<PID>.trc (incident=2776245):
ORA-29770: global enqueue process LMD1 (OSID <PID>) is hung for more than 70 seconds
Incident details in: $INCIDENT/incdir_2776245/<SID>_lmhb_<PID>_i2776245.trc
2017-09-30T10:33:51.417681+02:00
LOCK_DBGRP: GCR_SYSTEST debug event locked group GR+DB_<SID> by memno 1
ERROR: Some process(s) is not making progress.
LMHB (ospid: <PID>) is terminating the instance.
Please check LMHB trace file for more details.
Please also check the CPU load, I/O load and other system properties for anomalous behavior
ERROR: Some process(s) is not making progress.
2017-09-30T10:33:51.738764+02:00
LGWR waiting for instance termination
2017-09-30T10:33:51.756125+02:00
LMHB (ospid: <PID>): terminating the instance due to error 29770
2017-09-30T10:33:51.799736+02:00
opiodr aborting process unknown ospid (<PID>) as a result of ORA-1092
2017-09-30T10:33:51.866045+02:00
System state dump requested by (instance=1, oSID=<PID> (LMHB)), summary=[abnormal instance termination].
System State dumped to trace file $TRACE/<SID>_diag_<PID>_20170930103351.trc
2017-09-30T10:33:56.698260+02:00
License high water mark = 6085
2017-09-30T10:33:58.503461+02:00
Instance terminated by LMHB, pid = <PID>
2017-09-30T10:33:58.504368+02:00
Warning: 2 processes are still attach to shmid 1277964:
(size: 2097152 bytes, creator pid: 18501, last attach/detach pid: <PID>)
2017-09-30T10:33:58.698938+02:00
USER (ospid: <PID>): terminating the instance
2017-09-30T10:33:58.702239+02:00
Instance terminated by USER, pid = <PID>

 

*** 2017-09-30T10:33:45.868903+02:00

*** SESSION ID:(7802.10250) 2017-09-30T10:33:45.868924+02:00
*** CLIENT ID:() 2017-09-30T10:33:45.868927+02:00
*** SERVICE NAME:(SYS$BACKGROUND) 2017-09-30T10:33:45.868931+02:00
*** MODULE NAME:() 2017-09-30T10:33:45.868934+02:00
*** ACTION NAME:() 2017-09-30T10:33:45.868938+02:00
*** CLIENT DRIVER:() 2017-09-30T10:33:45.868940+02:00

[TOC00000]
Jump to table of contents
Dump continued from file: $TRACE/<SID>_lmhb_<PID>.trc
[TOC00001]
ORA-29770: global enqueue process LMD1 (OSID <PID>) is hung for more than 70 seconds

[TOC00001-END]
[TOC00002]
========= Dump for incident 2776245 (ORA 29770) ========
[TOC00003]
----- Beginning of Customized Incident Dump(s) -----
==================================================
=== LMD1 (ospid: <PID>) Heartbeat Report
==================================================
LMD1 (ospid: <PID>) has no heartbeats for 92 sec. (threshold 70)
: heartbeat state 0x1.ffff (inwait) pso-flag 0x100
: waiting for event 'SGA: allocation forcing component growth' for 92 secs with wait_id 247627611.
===[ Wait Chain ]===
LMD1 (ospid: <PID>) waits for event 'SGA: allocation forcing component growth'.
MMAN (ospid: 18695) is not in wait.
=[SNAPSHOT SUMMARY]=
Process did not not move according to collected snapshots
===[NO-HB PROCESS CALLSTACK]===
...__sighandler... ipcgxp_selectex <- ipclw_wait <- ksxpwait_ipclw <- ksxpwait_int <- ...

SO: 0x664b82ce0, type: 4, owner: 0x6a3b70150, flag: INIT/-/-/-/0x00 if: 0x3 c: 0x3
proc=0x6a3b70150, name=session, file=ksu.h LINE:15737, pg=0, conuid=0
(session) SID: 7264 ser: 18222 trans: (nil), creator: 0x6a3b70150
flags: (0x51) USR/- flags2: (0x409) -/-/INC
flags_idl: (0x1) status: BSY/-/-/- kill: -/-/-/-
DID: 0000-0000-000000000000-0000-00000000, short-term DID:
txn branch: (nil)
edition#: 0 user#/name: 0/SYS
oct: 0, prv: 0, sql: (nil), psql: (nil)
stats: 0x6dfe57fd8, PX stats: 0x1106b044
service name: SYS$BACKGROUND
Current Wait Stack:
0: waiting for 'SGA: allocation forcing component growth'
=0x0, =0x0, =0x0
wait_id=247627611 seq_num=37255 snap_id=1
wait times: snap=1 min 32 sec, exc=1 min 32 sec, total=1 min 32 sec                                              <============
wait times: max=infinite, heur=1 min 32 sec
wait counts: calls=9006 os=9006
in_wait=1 iflags=0x15a0
There is at least one session blocking this session.
Dumping 1 direct blocker(s):
inst: 1, SID: 2153, ser: 18977
Dumping final blocker:
inst: 1, SID: 2153, ser: 18977

Time: 2017-09-30 10:33:40.515270 +02:00

Wait Stack: waiting for event 'SGA: allocation forcing component growth' for 86 secs with wait_id 247627611.
Wait History:
0 : 'ges remote message' (waitid 247627610)
snap time=0.000983 sec total time=0.000983 sec
time since last wait=0.000050 sec
1 : 'ges remote message' (waitid 247627609)
snap time=0.000958 sec total time=0.000958 sec
time since last wait=0.000052 sec
2 : 'ges remote message' (waitid 247627608)
snap time=0.000806 sec total time=0.000806 sec
time since last wait=0.000053 sec
...

*** 2017-09-30T10:33:00.520877+02:00
Process diagnostic dump for oracle@l9930022 (LMD1), OS id=<PID>,
pid: 27, proc_ser: 1, SID: 7264, sess_ser: 18222
-------------------------------------------------------------------------------
os thread scheduling delay history: (sampling every 1.000000 secs)
0.000000 secs at [ 10:32:59 ]
NOTE: scheduling delay has not been sampled for 0.948332 secs
0.000000 secs from [ 10:32:55 - 10:33:00 ], 5 sec avg
0.000000 secs from [ 10:32:01 - 10:33:00 ], 1 min avg
0.000010 secs from [ 10:28:01 - 10:33:00 ], 5 min avg

...

F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD
0 S oracle <PID> 1 4 80 0 - 65657690 - Sep25 ? 04:36:57 ora_lmd1_<SID>
Name: ora_lmd1_<SID>
State: S (sleeping)

 

Error: ORA-29770 [LMD1] [<PID>] [70] [] [] [] [] [] [] [] [] []
dbgeEndDDEInvocationImpl [diag_dde]
kjfmHBgenerateADR [LMHB]                                                             <-- Signaling
kjfmGCR_VerifyNoHeartbeatProc [LMHB]
kjfmGCR_HBdisambig [LMHB]
kjgcr_ActionCheckMatch []
...