My Oracle Support Banner

Automatic Storage Management (ASM) Instance Shutdown Hanging Due to oraagent.bin Spinning (Doc ID 2230416.1)

Last updated on FEBRUARY 12, 2019

Applies to:

Oracle Database Cloud Schema Service - Version N/A and later
Oracle Database Exadata Cloud Machine - Version N/A and later
Oracle Database Exadata Express Cloud Service - Version N/A and later
Oracle Cloud Infrastructure - Database Service - Version N/A and later
Oracle Database Cloud Exadata Service - Version N/A and later
Information in this document applies to any platform.


 CRS is stopped using crsctl stop crs command and it takes long time and found the delay is due to  CKPT process on ASM instance is blocked by OSPID=27829(oraagent.bin) and ORA-29771 is returned.

When problem happened, server process (OSPID=27829) on ASM instance spins and consumes CPU resource. This process is  from oraagent.bin.

From ASM alert log:




NOTE: client is exiting
NOTE: client is exiting
NOTE: client is exiting
Errors in file <ASM trace directory>/+ASM1_lmhb_3665.trc
(incident=92954):ORA-29771: process USER (OSID 27829) blocks CKPT (OSID 3681) for more than 150 seconds=======>agent process spins
NOTE: client is exiting
NOTE: client is exiting
NOTE: client is exiting

OSPID=27829 spins and write above messages repeatedly.

In trace file of LMHB, this process is running and does not wait in some wait events (Current Wait Stack is "Not in wait").

< +ASM1_lmhb_3665.trc >

*** 2016-12-15 19:39:38.769
loadavg : 2.14 1.92 1.84 System user time: 0.04 sys time: 0.05 context switch: 12906
Memory (Avail / Total) = 10071.48M / 129003.43M
Swap (Avail / Total) = 130773.43M / 131071.99M
0 R 1004 27829 1 99 99 19 - 82920 - 19:36 ? 00:02:45 oracle+ASM1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))

SO: 0x8799b1f8, type: 4, owner: 0x87e06d68, flag: INIT/-/-/0x00
 proc=0x87e06d68, name=session, file=ksu.h LINE:13957, pg=0 conuid=0
(session) sid: 169 ser: 20588 trans: (nil), creator: 0x87e06d68
 flags: (0x841) USR/- flags_idl: (0x1) BSY/-/-/-/-/-
 flags2: (0x9) -/-/INC
 DID: 0000-001A-00002C68, short-term DID: 0000-0000-00000000
txn branch: (nil)
 edition#: 0 user#/name: 0/SYS
 oct: 0, prv: 0, sql: (nil), psql: (nil)
 stats: 0x8025ae88, PX stats: 0xbe34370
ksuxds FALSE at location: 0
ervice name: SYS$USERS
client details:
O/S info: user: oraclegrid, term: , ospid: 2581
machine: VZD001 program: oraagent.bin@VZD001 (TNS V1-V3)========>oraagent.bin process
application name: oraagent.bin@VZD001 (TNS V1-V3), hash value=1590433201
Current Wait Stack: Not in wait; last wait ended 2 min 47 sec ago==================>not in any wait
Wait State:
 fixed_waits=0 flags=0x20 boundary=(nil)/-1

CKPT is waiting for latch 'ASM db client latch'.OSPID=27829 holds this latch.

*** 2016-12-15 19:39:34.657
CKPT (ospid: 3681) has not moved for 162 sec (1481798374.1481798212)
: heartbeat check status 6 (no-heartbeat) (threshold 150 sec)
=== CKPT (ospid: 3681) Heartbeat Report
CKPT (ospid: 3681) has no heartbeats for 162 sec. (threshold 150)
: heartbeat state 0x5.ffff (inwait|latch-get) pso-flag 0x100
: waiting for event 'latch free' for 159 secs with wait_id 153014.
: waiting for latch 'ASM db client latch'.
===[ Wait Chain ]===
Wait chain is empty.
===[ Latch Chain ]===
CKPT (ospid: 3681) waits for latch 'ASM db client latch'.=========>CKPT wait
USER (ospid: 27829) is not waiting for any latch.========>agent is not in any wait
In trace file of OSPID=27829, "NOTE: client is exiting" messages appears===>client exiting messages
repeatedly (we see same message for 2.3 million lines).



In +ASM1_lmhb_3665.trc, there are 2 stack info for OSPID=27829.

Stack is similar, but not completely same.dbgl and dbkr function writes info in files.These functions are called from kfnCanShutdown().
In kfnCanShutdown(), 'ASM db client latch' is gotten and OSPID=27829 holds this latch when problem happened.




To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!

In this Document

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.