My Oracle Support Banner

ORA-29770: global enqueue process LGWR (ospid: xxxxxx) is hung for more than 70 seconds (Doc ID 2457765.1)

Last updated on MARCH 08, 2019

Applies to:

Oracle Database - Enterprise Edition - Version 12.1.0.2 to 12.1.0.2 [Release 12.1]
Information in this document applies to any platform.

Symptoms

12.1.0.2 Real Application Cluster (RAC) instance terminates with following message:

alert_RACDB1.log from <oracle base>/diag/rdbms/<dbname>/<instance name>/alert_<instance name>.log location:

2018-10-08 21:57:54.511000 -04:00
LGWR (ospid: 350888) has not called a wait for 2 secs.
LGWR (ospid: 350888) is hung in an acceptable location (cfio 0x12.00).
2018-10-08 21:58:18.420000 -04:00
LGWR (ospid: 350888) has not called a wait for 5 secs.
2018-10-08 21:58:21.864000 -04:00
Errors in file /u01/app/oracle/diag/rdbms/RACDB/RACDB1/trace/RACDB1_lmhb_350864.trc (incident=2304197):
ORA-29770: global enqueue process LGWR (OSID 350888) is hung for more than 70 seconds
Incident details in: /u01/app/oracle/diag/rdbms/RACDB/RACDB1/incident/incdir_2304197/RACDB1_lmhb_350864_i2304197.trc
2018-10-08 21:58:27.817000 -04:00
LOCK_DBGRP: GCR_SYSTEST debug event locked group GR+<diskgroup> by memno 1
ERROR: Some process(s) is not making progress.
LMHB (ospid: 350864) is terminating the instance.
Please check LMHB trace file for more details.
Please also check the CPU load, I/O load and other system properties for anomalous behavior
ERROR: Some process(s) is not making progress.
LMHB (ospid: 350864): terminating the instance due to error 29770
System state dump requested by (instance=1, osid=350864 (LMHB)), summary=[abnormal instance termination].
System State dumped to trace file /u01/app/oracle/diag/rdbms/RACDB/RACDB1/trace/RACDB1_diag_350822_20181008215828.trc
2018-10-08 21:58:29.683000 -04:00
License high water mark = 174
2018-10-08 21:58:32.725000 -04:00
ORA-29770: global enqueue process (OSID ) is hung for more than seconds
Errors in file /u01/app/oracle/diag/rdbms/RACDB/RACDB1/trace/RACDB1_diag_350822_20181008215828.trc:
ORA-29770: global enqueue process (OSID ) is hung for more than seconds
Dumping diagnostic data in directory=[cdmp_20181008215828], requested by (instance=1, osid=350864 (LMHB)), summary=[abnormal instance termination].
2018-10-08 21:58:37.227000 -04:00
Termination issued to instance processes. Waiting for the processes to exit
Mon Oct 08 21:58:49 2018
ORA-1092 : opitsk aborting process
Instance termination failed to kill one or more processes
Mon Oct 08 21:58:53 2018
Instance terminated by LMHB, pid = 350864
Mon Oct 08 21:58:53 2018
USER (ospid: 388922): terminating the instance
Termination issued to instance processes. Waiting for the processes to exit
Mon Oct 08 21:59:03 2018
Instance termination failed to kill one or more processes
Termination issued to instance processes. Waiting for the processes to exit
Mon Oct 08 21:59:13 2018
Instance termination failed to kill one or more processes
Mon Oct 08 21:59:13 2018
Instance terminated by USER, pid = 388922
Mon Oct 08 21:59:21 2018
Starting ORACLE instance (normal) (OS id: 390703)
Mon Oct 08 21:59:21 2018

lmhb trace has following messages

/u01/app/oracle/diag/rdbms/RACDB/RACDB1/trace/RACDB1_lmhb_350864.trc:

*** 2018-10-08 21:58:21.768
==============================
LGWR (ospid: 350888) has not moved for 113 sec (1539050301.1539050188)
  : wait in cfio is not acceptable as it fails the global check.
Incident 2304197 created, dump file: /u01/app/oracle/diag/rdbms/RACDB/RACDB1/incident/incdir_2304197/RACDB1_lmhb_350864_i2304197.trc
ORA-29770: global enqueue process LGWR (OSID 350888) is hung for more than 70 seconds

kjfmGCR_HBdisambig: action=Inst-kill

*** 2018-10-08 21:58:27.666
kjgcr_DoPreAction: KJGCR_PREACTION? - id 1

*** 2018-10-08 21:58:27.816
GCR_SYS group GR+<diskgroup> LOCKED by memno 1

*** 2018-10-08 21:58:27.816
LOCK_DBGRP: GCR_SYSTEST debug event locked group GR+<diskgroup> by memno 1

*** 2018-10-08 21:58:28.122
kjgcr_DoAction: KJGCR_ACTION? - id 1

*** 2018-10-08 21:58:28.122
==============================
LGWR (ospid: 350888) has not moved for 119 sec (1539050307.1539050188)
  : wait in cfio is not acceptable as it fails the global check.
2018-10-08 21:58:28.127249 :kjzduptcctx(): Notifying DIAG for crash event
----- Abridged Call Stack Trace -----
ksedsts()+244<-kjzdssdmp()+321<-kjzduptcctx()+692<-kjzdicrshnfy()+992<-ksuitm()+5877<-kjgcr_KillInstance()+150<-kjgcr_RunCallback()+330<-kjgcr_DoAction()+162<-kjgcr_servicecssg()+861<-kjgcr_Main()+370<-kjfmlmhb_Main()+446<-ksbrdp()+1072<-opirip()+1488<-opidrv()+616
<-sou2o()+145<-opimai_real()+270<-ssthrdmain()+412<-main()+236<-__libc_start_main()+253
----- End of Abridged Call Stack Trace -----

 

LMHB Incident trace has following message.

/u01/app/oracle/diag/rdbms/RACDB/RACDB/incident/incdir_2304197/RACDB1_lmhb_350864_i2304197.trc:


*** 2018-10-07 04:47:20.384
*** SESSION ID:(3478.7616) 2018-10-07 04:47:20.384
*** CLIENT ID:() 2018-10-07 04:47:20.384
*** SERVICE NAME:() 2018-10-07 04:47:20.384
*** MODULE NAME:() 2018-10-07 04:47:20.384
*** CLIENT DRIVER:() 2018-10-07 04:47:20.384
*** ACTION NAME:() 2018-10-07 04:47:20.384


ORA-29770: global enqueue process LGWR (OSID 350888) is hung for more than 70
seconds

LGWR (ospid: 350888) has no heartbeats for 84 sec. (threshold 70)
  : heartbeat state 0x11.00 (cfio) pso-flag 0x0
  : waiting for event 'control file parallel write' for 0 secs with wait_id
11755069.
===[ Wait Chain ]===
Wait chain is empty  <<<
==============================
Dumping PROCESS LGWR (ospid: 350888) States
==============================
===[ System Load State ]===
  CPU Total 16 Raw 16 Core 16 Socket 1
  Load normal: Cur 2042 Highmark 20480 (7.97 80.00)
===[ Latch State ]===
  Not in Latch Get    <<<<<<<<<

Changes

 

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


In this Document
Symptoms
Changes
Cause
Solution
 Community Discussions
References


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.