LMHB terminating with error 29770 as LMON not calling a wait with kjdrbnftpoll in call stack (Doc ID 2251012.1)

Mon Jan 23 11:40:50 2017
LMON (ospid: 92325) has not called a wait for 84 secs.
Errors in file /dborafiles/admin/diag/rdbms/<test>/<test4>/trace/<test4>_lmhb_93093.trc (incident=7600566) (PDBNAME=CDB$ROOT):
ORA-29770: global enqueue process LMON (OSID 92325) is hung for more than 70 seconds : global enqueue process LMON (OSID 92325) is hung for more than 70 seconds

Incident details in: /dborafiles/admin/diag/rdbms/<test>/<test4>/incident/incdir_7600566/<test4>_lmhb_93093_i7600566.trc
LOCK_DBGRP: GCR_SYSTEST debug event locked group GR+DB_EDW01_TP by memno 0
ERROR: Some process(s) is not making progress.
LMHB (ospid: 93093) is terminating the instance.

Mon Jan 23 11:40:54 2017
LMON (ospid: 746) has not called a wait for 89 secs.
Errors in file /dborafiles/admin/diag/rdbms/test/<test2>/trace/<test2>_lmhb_1449.trc (incident=8327345) (PDBNAME=CDB$ROOT):
ORA-29770: global enqueue process LMON (OSID 746) is hung for more than 70 seconds : global enqueue process LMON (OSID 746) is hung for more than 70 seconds
Incident details in: /dborafiles/admin/diag/rdbms/<test>/<test2>/incident/incdir_8327345/<test2>_lmhb_1449_i8327345.trc
Mon Jan 23 11:40:57 2017

DDE: Problem Key 'ORA 29770' was completely flood controlled (0x6)
Further messages for this problem key will be suppressed for up to 10 minutes
Mon Jan 23 11:44:21 2017

Mon Jan 23 11:44:35 2017
diag_adl:IPC Send timeout to 5.1 inc 34 for msg type 65521 from opid 721
Mon Jan 23 11:44:44 2017
LMS0 (ospid: 882) received an instance eviction notification from instance 1 [2]
Mon Jan 23 11:44:47 2017
Instance Critical Process (pid: 14, ospid: 746, LMON) died unexpectedly
PMON (ospid: 98145): terminating the instance due to error 481
Mon Jan 23 11:44:47 2017
System state dump requested by (instance=2, osid=98145 (PMON)), summary=[abnormal instance termination].
System State dumped to trace file /dborafiles/admin/diag/rdbms/<test>/<test2>/trace/<test2>_diag_593_20170123114447.trc

[TOC00000]

Jump to table of contents
Dump continued from file: /dborafiles/admin/diag/rdbms/<test>/<test2>/trace/<test2>_lmhb_1449.trc
[TOC00001]
ORA-29770: global enqueue process LMON (OSID 746) is hung for more than 70 seconds

[TOC00001-END]
[TOC00002]
========= Dump for incident 8327348 (ORA 29770) ========
[TOC00003]
----- Beginning of Customized Incident Dump(s) -----
==================================================
=== LMON (ospid: 746) Heartbeat Report
==================================================
LMON (ospid: 746) has no heartbeats for 106 sec. (threshold 70)
: heartbeat state 0x2.ffff () pso-flag 0x100
: Not in wait; last wait ended 101 secs ago.
: last wait_id 184369031 at 'ges lms sync during dynamic remastering and reconfig'.

*** 2017-01-23 11:44:48.088
PROCESS 14: LMON
----------------------------------------
SO: 0x2012db51c0, type: 2, owner: (nil), flag: INIT/-/-/0x00 if: 0x3 c: 0x3
proc=0x2012db51c0, name=process, file=ksu.h LINE:13949, pg=0 conuid=0
(process) Oracle pid:14, ser:1, calls cur/top: 0x204442cd88/0x204442cd88
flags : (0x6) SYSTEM icon_uid:0
flags2: (0x100), flags3: (0x10)
intr error: 0, call error: 0, sess error: 0, txn error 0
intr queue: empty
ksudlp FALSE at location: 0
(post info) last post received: 1092 256 12
last post received-location: ksu.h LINE:15411 ID:ksusig
last process to post me: 0x1ff2ca5778 1 14
last post sent: 0 0 93
last post sent-location: kji.h LINE:4733 ID:kjfclmsync: unfreeze LMS in parallel rcfg sync
last process posted by me: 0x2002c97fd0 1 6
waiter on post event: 0
(latch info) hold_bits=0x64
Holding:
0x205a6a4f40 Child 'gcs partitioned table hash' (level=6, child#=7)
Holder Location: kjdr.h LINE:2315 ID:kjdrbnftpoll no wait mode, single pkey stats:
Holder Slot: slot=6, efd=4, pdb=1
Holder Context: 0
Latch State:
state=busy (1 shared) [value=0x1]
holder orapid=14, ospid=746
wlstate=free [value=0]
Holding:
0x1de0d2b178 Child 'process pkey statistics list' (level=5, child#=14)
Holder Location: kjp.h LINE:1386 ID:kjdrbnftpoll: updating single pkey:
Holder Slot: slot=5, efd=4, pdb=1
Holder Context: 0
Latch State:
state=busy (exclusive) [value=0x200000000000000e]
holder orapid=14, ospid=746
wlstate=free [value=0]
Holding:
0x20060a7d38 Child 'ges process hash list' (level=2, child#=89)
Holder Location: kjp.h LINE:1367 ID:kjdrbnftpoll: benefit statistics polling:
Holder Slot: slot=2, efd=4, pdb=1
Holder Context: 0
Latch State:
state=busy [value=0xe]
holder orapid=14, ospid=746

 

Short stack dump:

kjdrbnftpoll()+3073<-kjfcrfg()+97062<-kjfcln()+10851<-ksbrdp()+1068<-opirip()