My Oracle Support Banner

ORA-29770: global enqueue process LCK0 (OSID xxxx) is hung for more than 70 seconds (Doc ID 2369507.1)

Last updated on AUGUST 04, 2018

Applies to:

Oracle Database - Enterprise Edition - Version 12.1.0.2 and later
Information in this document applies to any platform.

Symptoms

 12.1.0.2 RAC instance crashes:

alert.log

Tue Feb 20 19:30:05 2018
Archived Log entry 16908 added for thread 1 sequence 216737 ID 0xb00cd7a dest 1:
Tue Feb 20 19:35:07 2018
Completed checkpoint up to RBA [0x34ea2.2.10], SCN: 729099343422
Tue Feb 20 20:00:05 2018
Beginning log switch checkpoint up to RBA [0x34ea3.2.10], SCN: 729099838007
Tue Feb 20 20:00:05 2018
Thread 1 advanced to log sequence 216739 (LGWR switch)
Current log# 102 seq# 216739 mem# 0: +ASMxxx/xxx/ONLINELOG/group_102.284.944745475
Tue Feb 20 20:00:05 2018
Archived Log entry 16909 added for thread 1 sequence 216738 ID 0xb00cd7a dest 1:
Tue Feb 20 20:05:07 2018
Completed checkpoint up to RBA [0x34ea3.2.10], SCN: 729099838007
Tue Feb 20 20:11:20 2018
Resize operation completed for file# 868, old size 10485760K, new size 11010048K
Resize operation completed for file# 866, old size 11534336K, new size 12058624K
Tue Feb 20 20:18:08 2018
LCK0 (ospid: xxxxx) waits for event 'rdbms ipc message' for 76 secs.
Errors in file /u01/app/oracle/diag/rdbms/xxx/xxxx_1/trace/xxx_1_lmhb_33058.trc (incident=1115787):
ORA-29770: global enqueue process LCK0 (OSID xxxxx) is hung for more than 70 seconds
Incident details in: /u01/app/oracle/diag/rdbms/xxx/xxx_1/incident/incdir_1115787/xxx_1_lmhb_xxxx_i1115787.trc
Tue Feb 20 20:18:29 2018
Sweep [inc][1115787]: completed
Sweep [inc2][1115787]: completed
Tue Feb 20 20:18:29 2018
LOCK_DBGRP: GCR_SYSTEST debug event locked group GR+DB_xxx by memno 0
ERROR: Some process(s) is not making progress.
LMHB (ospid: xxxx) is terminating the instance.
Please check LMHB trace file for more details.
Please also check the CPU load, I/O load and other system properties for anomalous behavior
ERROR: Some process(s) is not making progress.
LMHB (ospid: xxxx): terminating the instance due to error 29770


lmhb trace

Dump file /u01/app/oracle/diag/rdbms/xxx/xxx_1/incident/incdir_1115787/xxxx_1_lmhb_xxxx_i1115787.trc
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Advanced Analytics, Real Application Testing and Unified Auditing options
ORACLE_HOME = /u01/app/oracle/product/12.1.0/db_2
System name: Linux
Node name: xxxx.com
Release: 4.1.12-37.4.1.el6uek.x86_64
Version: #2 SMP Tue May 17 07:23:38 PDT 2016
Machine: x86_64
Instance name: xxxx_1
Redo thread mounted by this instance: 1
Oracle process number: 20
Unix process pid: xxxx, image: oracle@xxxx.com (LMHB)

*** 2018-02-20 20:18:18.782
*** SESSION ID:(758.41652) 2018-02-20 20:18:18.782
*** CLIENT ID:() 2018-02-20 20:18:18.782
*** SERVICE NAME:(SYS$BACKGROUND) 2018-02-20 20:18:18.782
*** MODULE NAME:() 2018-02-20 20:18:18.782
*** CLIENT DRIVER:() 2018-02-20 20:18:18.782
*** ACTION NAME:() 2018-02-20 20:18:18.782

[TOC00000]
Jump to table of contents
Dump continued from file: /u01/app/oracle/diag/rdbms/xxx/xxx_1/trace/xxx_1_lmhb_xxxx.trc
[TOC00001]
ORA-29770: global enqueue process LCK0 (OSID xxxx) is hung for more than 70 seconds

[TOC00001-END]
[TOC00002]
========= Dump for incident 1115787 (ORA 29770) ========
[TOC00003]
----- Beginning of Customized Incident Dump(s) -----
==================================================
=== LCK0 (ospid: 33581) Heartbeat Report
==================================================
LCK0 (ospid: 33581) has no heartbeats for 92 sec. (threshold 70)
: heartbeat state 0x1.ffff (inwait) pso-flag 0x0
: waiting for event 'rdbms ipc message' for 86 secs with wait_id 999040302.
===[ Wait Chain ]===
Wait chain is empty.
==============================
Dumping PROCESS LCK0 (ospid: xxxx) States
==============================
===[ System Load State ]===
CPU Total 16 Raw 16 Core 8 Socket 2
Load normal: Cur 1018 Highmark 20480 (3.97 80.00)
===[ Latch State ]===
Not in Latch Get
===[ Session State Object ]===
----------------------------------------
SO: 0xed18597d8, type: 4, owner: 0xed9180288, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
proc=0xed9180288, name=session, file=ksu.h LINE:13957, pg=0 conuid=0
(session) sid: 1704 ser: 58614 trans: (nil), creator: 0xed9180288
flags: (0x51) USR/- flags_idl: (0x1) BSY/-/-/-/-/-
flags2: (0x409) -/-/INC
DID: 0000-0000-00000000, short-term DID: 0000-0000-00000000
txn branch: (nil)
edition#: 0 user#/name: 0/SYS
oct: 0, prv: 0, sql: (nil), psql: (nil)
stats: 0xe2ffe61d0, PX stats: 0xd2afbf0
ksuxds FALSE at location: 0
service name: SYS$BACKGROUND
Current Wait Stack:
1: waiting for 'CSS group membership query'
group_name1=0x49474d41, group_name2=0x524b5055, group_name3=0x52455359
wait_id=999040438 seq_num=25167 snap_id=1
wait times: snap=1 min 23 sec, exc=1 min 23 sec, total=1 min 23 sec
wait times: max=infinite, heur=1 min 23 sec
wait counts: calls=0 os=0
in_wait=1 iflags=0x520
0: waiting for 'rdbms ipc message'
timeout=0x12c, =0x0, =0x0
wait_id=999040302 seq_num=25166 snap_id=136
wait times: snap=0.000000 sec, exc=3.299165 sec, total=1 min 26 sec
wait times: max=3.000000 sec, heur=1 min 26 sec
wait counts: calls=1 os=1
in_wait=1 iflags=0x15a8
Wait State:
fixed_waits=0 flags=0x2a boundary=0xed185a8c0/1
Session Wait History:
elapsed time of 0.000000 sec since current wait
0: waited for 'rdbms ipc message'
timeout=0x12c, =0x0, =0x0
wait_id=999040302 seq_num=25166 snap_id=136
wait times: snap=0.000001 sec, exc=3.299165 sec, total=3.364194 sec
wait times: max=3.000000 sec
wait counts: calls=1 os=1
occurred after 0.000000 sec of elapsed time
1: waited for 'CSS group membership query'
group_name1=0x49474d41, group_name2=0x524b5055, group_name3=0x52455359
wait_id=999040437 seq_num=25165 snap_id=1
wait times: snap=0.000393 sec, exc=0.000393 sec, total=0.000393 sec
wait times: max=infinite
wait counts: calls=0 os=0
occurred after 0.000000 sec of elapsed time
.....
........
----- Incident Context Dump -----
Address: 0x7ffe14a84e08
Incident ID: 1115787
Problem Key: ORA 29770
Error: ORA-29770 [LCK0] [33581] [70] [] [] [] [] [] [] [] [] []
[00]: dbgexExplicitEndInc [diag_dde]
[01]: dbgeEndDDEInvocationImpl [diag_dde]
[02]: dbgePostErrorKGE [diag_dde]
[03]: dbkePostKGE_kgsf [rdbms_dde]
[04]: kgesev []
[05]: ksesec3 [KSE]
[06]: ksdx_cmdreq_wait_for_pending [VOS]<-- Signaling
[07]: ksdxdocmdmultex [VOS]
[08]: ksdxdocmdmult [VOS]
[09]: ksudmp_proc_short_stack [ksu]
[10]: ksdhng_diag_proc_int [hang_analysis]
[11]: ksdhng_diag_proc [hang_analysis]
[12]: kjdgpstack_wTimeout []
[13]: kjfmHealth_Proc_Dump []
[14]: kjfmHBgenerateReport_i []
[15]: kjfmHBgenerateADR []
[16]: kjfmGCR_VerifyNoHeartbeatProc []
[17]: kjfmGCR_HBdisambig []
[18]: kjgcr_ActionCheckMatch []
[19]: kjgcr_ServiceGCR []
[20]: kjgcr_Main []
[21]: kjfmlmhb_Main []
[22]: ksbrdp [background_proc]
[23]: opirip [OPI]
[24]: opidrv [OPI]
[25]: sou2o []
[26]: opimai_real [OPI]
[27]: ssthrdmain []
[28]: main []
[29]: __libc_start_main []
MD [00]: 'SID'='758.41652' (0x2)
MD [01]: 'ProcId'='20.1' (0x2)
MD [02]: 'Service'='SYS$BACKGROUND' (0x200)
MD [03]: 'Client ProcId'='oracle@extduborclc803.eu.dsext.susq.com.33058_140189424824928' (0x0)
Impact 0:
Impact 1:
Impact 2:
Impact 3:
Derived Impact:
----- END Incident Context Dump ------

 

Running the following will dump the ksim PGA nd SGA cache
SQL> oradebug setorapname lck0
SQL> oradebug call ksimPrintGroupMemPGA
SQL> oradebug call ksimPrintGroupMemCache

If you check the LCK0 trace file after the above you may
see many datapump/AQ generated groups left behind in the
form of:
SYS.KUPC$S_<thread>_yyyymmddhhmiss
SYS.KUPC$C_<thread>_yyyymmddhhmiss

trace> grep -i "SYS.KUPC" xxx_1_lck0_95570.trc
[00] : 0x11000000 : 0 : 1 : 22 : IGMARKPURESYS.KUPC$C_1_20180221010003
[01] : 0x11000000 : 0 : 1 : 14 : IGMARKPURESYS.KUPC$S_1_20180221010003
c[14] : 1 : 0 : 1 : IGMARKPURESYS.KUPC$S_1_20180221010003
c[22] : 1 : 0 : 1 : IGMARKPURESYS.KUPC$C_1_20180221010003

 

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


In this Document
Symptoms
Cause
Solution
References


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.