My Oracle Support Banner

Many Errors Around LGWR Waiting On LATCH FREE AND TMON process Blocking LGWR and Other processes In Diag Trace files (Doc ID 2929094.1)

Last updated on MAY 17, 2023

Applies to:

Oracle Database - Enterprise Edition - Version 19.13.0.0.0 to Beta 23c [Release 19 to 21.0]
Information in this document applies to any platform.

Symptoms

On : 19.13.0.0.0 version, Exa-Oracle Data Guard

On Primary there are Many errors around LGWR waiting on a LATCH FREE and TMON process is the Final blocker in the diag trace files.

Redo transport will show ORA-16198 and other transport errors including ARC process waiting on  'Redo Transport Detach'.

 

The LMHB process will automatically kill the TMON process after some time which will correct the issue but the Database is Hung until TMON is killed.

 

Primary Alert log symptoms:

==================

LGWR (ospid: XXXX) waits for event 'latch free' for 81 secs.
LGWR (ospid: XXXX) waits for latch 'archive process' for 81 secs.
2022-11-02T09:44:29.913231-07:00
Process termination requested for pid XXXX [source = rdbms], [info = 2]
[request issued by pid: XXXX, uid: 1000]
2022-11-02T09:44:39.883116-07:00
Errors in file
/u01/app/oracle/diag/rdbms/XXXX_XXXX/XXXX/trace/XXXX_lmhb_XXXX
.trc (incident=XXXX):
ORA-29771: process TMON (OSID XXXX) blocks LGWR (OSID XXXX) for more than
70 seconds
Incident details in:
/u01/app/oracle/diag/rdbms/XXXX_XXXX/XXXX/incident/incdir_XXXX/pr
01pimi1_lmhb_XXXX_iXXXX.trc
2022-11-02T09:44:44.763527-07:00
TMON (ospid: XXXX) is blocking LGWR (ospid: XXXX) in a wait
LMHB (ospid: XXXX) kills TMON (ospid: XXXX).

 

 

The  "xxxx_dia0_XXXX_lws_1.trc" shows TMON is the Final Blocker:
============================================
 


*** 2022-11-02T09:43:08.554845-07:00
HM: Early Warning - Session ID 2114 serial# XXXX OS PID XXXX (TMON) is
'not in a wait' for 32 seconds
last waited on 'rdbms ipc message'
p1: 'timeout'=0xb1f, p2: ''=0x0, p3: ''=0x0
Session ID 2114 is blocking 5 sessions
Blocking Session ID XXXX serial# XXXX on instance 8
which is waiting on 'latch free' for 11 seconds
p1: 'address'=0x6006d000, p2: 'number'=0x169, p3: 'why'=0x0

IO
Total Self- Total Total Outlr Outlr Outlr
Hung Rslvd Rslvd Wait WaitTm Wait WaitTm Wait
Sess Hangs Hangs Count Secs Count Secs Count Wait Event
------ ------ ------ ------ ------ ------ ------ ------ -----------
2 0 0 0 0 0 0 0 not in wait

HM: Dumping Short Stack of pid[108.130914] (sid:2114, ser#:2371)
Short stack dump:
ksedsts()+426<-ksdxfstk()+58<-ksdxcb()+872<-sspuser()+200<-__sighandler()<-__n
an
osleep()+16<-krsh_time_sleep()+161<-krso_proc_set_state()+1270<-krso_proc_chec
k(
)+493<-krso_check_all()+848<-krso_tmon_wakeup()+1388<-ksb_act_run_int()+117<-k
sb
_act_run()+130<-ksbabs()+746<-ksbrdp()+1167<-opirip()+541<-opidrv()+581<-sou2o
()
+165<-opimai_real()+173<-ssthrdmain()+417<-main()+256<-__libc_start_main()+245

And see ARC0 is blocked by TMON:

2022-11-02T09:43:28.852266-07:00
HM: Early Warning - Session ID 2403 serial# XXXX OS PID XXXX (ARC0)
is waiting on 'latch free' for 32 seconds, wait id 714098
p1: 'address'=0x6006d000, p2: 'number'=0x169, p3: 'why'=0x0
Final Blocker is Session ID 2114 serial# 2371 on instance 8 << TMON blocking ARC0 as Final Blocker
which is 'not in a wait' for 52 seconds

IO
Total Self- Total Total Outlr Outlr Outlr
Hung Rslvd Rslvd Wait WaitTm Wait WaitTm Wait
Sess Hangs Hangs Count Secs Count Secs Count Wait Event
------ ------ ------ ------ ------ ------ ------ ------ -----------
0 0 0 39496953 8285 0 0 0 latch free

HM: Dumping Short Stack of pid[XXX.XXXX] (sid:2403, ser#:XXXX)
Short stack dump:
ksedsts()+426<-ksdxfstk()+58<-ksdxcb()+872<-sspuser()+200<-__sighandler()<-sem
op
()+7<-skgpwwait()+187<-kslges()+1525<-kslgetl()+2745<-krsa_latch_get()+186<-kr
ss
_proc_role()+126<-kcrrwkx()+242<-kcrrwk()+470<-ksb_act_run_int()+117<-ksb_act_
ru
n()+130<-ksbcti()+1862<-ksbabs()+2362<-ksbrdp()+1167<-opirip()+541<-opidrv()+5
81
<-sou2o()+165<-opimai_real()+173<-ssthrdmain()+417<-main()+256<-__libc_start_m
ai
n()+245

 

Changes

 

SQL> alter system set log_archive_max_processes=1 scope=both sid='*' ; and then back to original value

 

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!

In this Document
Symptoms
Changes
Cause
Solution
References


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.