ASM instance shutdown hangs due to oraagent.bin spins (Doc ID 2230416.1)

Last updated on FEBRUARY 08, 2017

Applies to:

Oracle Database - Enterprise Edition - Version 12.1.0.2 to 12.1.0.2 [Release 12.1]
Information in this document applies to any platform.

Symptoms

 CRS is stopped using crsctl stop crs command and it takes long time and found the delay is due to  CKPT process on ASM instance is blocked by OSPID=27829(oraagent.bin) and ORA-29771 is returned.
.


When problem happened, server process (OSPID=27829) on ASM instance spins and consumes CPU resource. This process is  from oraagent.bin.

From ASM alert log:

==============

[alert_+ASM1.log]

.

NOTE: client is exiting
NOTE: client is exiting
NOTE: client is exiting
Errors in file /var/log/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_lmhb_3665.trc
(incident=92954):ORA-29771: process USER (OSID 27829) blocks CKPT (OSID 3681) for more than 150 seconds=======>agent process spins
NOTE: client is exiting
NOTE: client is exiting
NOTE: client is exiting

OSPID=27829 spins and write above messages repeatedly.


.
In trace file of LMHB, this process is running and does not wait in some wait events (Current Wait Stack is "Not in wait").
.
============================================

< +ASM1_lmhb_3665.trc >

*** 2016-12-15 19:39:38.769
loadavg : 2.14 1.92 1.84 System user time: 0.04 sys time: 0.05 context switch: 12906
Memory (Avail / Total) = 10071.48M / 129003.43M
Swap (Avail / Total) = 130773.43M / 131071.99M
 S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD
0 R 1004 27829 1 99 99 19 - 82920 - 19:36 ? 00:02:45 oracle+ASM1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))


----------------------------------------
SO: 0x8799b1f8, type: 4, owner: 0x87e06d68, flag: INIT/-/-/0x00
 proc=0x87e06d68, name=session, file=ksu.h LINE:13957, pg=0 conuid=0
(session) sid: 169 ser: 20588 trans: (nil), creator: 0x87e06d68
 flags: (0x841) USR/- flags_idl: (0x1) BSY/-/-/-/-/-
 flags2: (0x9) -/-/INC
 DID: 0000-001A-00002C68, short-term DID: 0000-0000-00000000
txn branch: (nil)
 edition#: 0 user#/name: 0/SYS
 oct: 0, prv: 0, sql: (nil), psql: (nil)
 stats: 0x8025ae88, PX stats: 0xbe34370
ksuxds FALSE at location: 0
ervice name: SYS$USERS
client details:
O/S info: user: oraclegrid, term: , ospid: 2581
machine: VZD001 program: oraagent.bin@VZD001 (TNS V1-V3)========>oraagent.bin process
application name: oraagent.bin@VZD001 (TNS V1-V3), hash value=1590433201
Current Wait Stack: Not in wait; last wait ended 2 min 47 sec ago==================>not in any wait
Wait State:
 fixed_waits=0 flags=0x20 boundary=(nil)/-1


.
CKPT is waiting for latch 'ASM db client latch'.OSPID=27829 holds this latch.
.

====================================
*** 2016-12-15 19:39:34.657
==============================
CKPT (ospid: 3681) has not moved for 162 sec (1481798374.1481798212)
: heartbeat check status 6 (no-heartbeat) (threshold 150 sec)
==================================================
=== CKPT (ospid: 3681) Heartbeat Report
==================================================
CKPT (ospid: 3681) has no heartbeats for 162 sec. (threshold 150)
: heartbeat state 0x5.ffff (inwait|latch-get) pso-flag 0x100
: waiting for event 'latch free' for 159 secs with wait_id 153014.
: waiting for latch 'ASM db client latch'.
===[ Wait Chain ]===
Wait chain is empty.
===[ Latch Chain ]===
CKPT (ospid: 3681) waits for latch 'ASM db client latch'.=========>CKPT wait
USER (ospid: 27829) is not waiting for any latch.========>agent is not in any wait
====================================
.
In trace file of OSPID=27829, "NOTE: client is exiting" messages appears===>client exiting messages
repeatedly (we see same message for 2.3 million lines).
.

====================================

 

In +ASM1_lmhb_3665.trc, there are 2 stack info for OSPID=27829.
.

.
Stack is similar, but not completely same.dbgl and dbkr function writes info in files.These functions are called from kfnCanShutdown().
.
In kfnCanShutdown(), 'ASM db client latch' is gotten and OSPID=27829 holds this latch when problem happened.

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms