ASM Hang:: ASM Rebalance Process (ARBn) Blocked by ASM CKPT
(Doc ID 1565248.1)
Last updated on DECEMBER 15, 2022
Applies to:
Oracle Database - Enterprise Edition - Version 11.2.0.3 and laterOracle Database Cloud Schema Service - Version N/A and later
Gen 1 Exadata Cloud at Customer (Oracle Exadata Database Cloud Machine) - Version N/A and later
Oracle Cloud Infrastructure - Database Service - Version N/A and later
Oracle Database Cloud Exadata Service - Version N/A and later
Information in this document applies to any platform.
Symptoms
The rebalance operation for DATA and RECO are running on nodes 1 and 6.
Rebalance process is hanged.Database is up and running ,properly operation.
Pstack of asm_rbal process looks like below,
#4 0x0000000007f823db in kslwaitctx ()
#5 0x000000000425069b in kjusuc ()
#6 0x00000000063343eb in ksigeti ()
#7 0x000000000215cea6 in ksbcic_int ()
#8 0x0000000002157c56 in ksbcic ()
#9 0x0000000003d4571c in kffMapLock ()
#10 0x0000000003d45496 in kffRelocateMapLock ()
#11 0x0000000003d4f605 in kffRelocateClientData ()
#12 0x0000000003d72a73 in kffRelocateSubmit ()
#13 0x0000000003d73174 in kffRelocate ()
#14 0x0000000003cf4026 in kfdaExecute ()
#15 0x0000000003cc3d29 in kfdCompact ()
#16 0x0000000003cc4cc2 in kfdExecute ()
#17 0x0000000003dc4dda in kfgbRebalExecute ()
#18 0x0000000003db24f3 in kfgbDriver ()
#19 0x0000000002160345 in ksbabs ()
#20 0x0000000003dc897a in kfgbRun ()
Oracle session identified by:
{
instance: 4 (+asm.+asm4)
os id: 2321
process id: 51, oracle@xxx.xxx.xxx.com (TNS V1-V3)
session id: 198
session serial #: 26865
}
is waiting for 'DFS lock handle' with wait info:
{
p1: 'type|mode'=0x43490005
p2: 'id1'=0x2b
p3: 'id2'=0x5
time in wait: 37 min 40 sec
timeout after: never
wait id: 42
blocking: 0 sessions
current sql: alter diskgroup RECO_USP1 rebalance power 32
short stack: ksedsts()+461<-ksdxfstk()+32<-ksdxcb()+1876<-sspuser()+112<-__sighandler()<-semtimedop()+10<-skgpwwait()+160<-ksliwat()+1865<-kslwaitctx()+163<-kjusuc()+3619<-ksigeti()+3781<-ksbcic_int()+5092<-ksbcicr()+207<-kfnmGroupBlockGlobal()+2216<-kfnmGroupBlockPriv()+282<-kfgFinalize()+339<-kfxdrvAlter()+3265<-kfxdrvEntry()+1306<-opiexe()+20165<-opiosq0()+3870<-kpooprx()+274<-kpoal8()+800<-opiodr()+916<-ttcpip()+2242<-opitsk()+1673<-opiino()+966<-opiodr()+916<-opidrv()+570<-sou2o()+103<-opimai_real()+133<-ssthrdmain()+252<
wait history:
* time between current wait and wait #1: 0.000000 sec
1. event: 'ASM file metadata operation'
time waited: 0.000038 sec (last interval)
time waited: 0.013707 sec (total)
wait id: 40 p1: 'msgop'=0x0
p2: 'locn'=0x4
* time between wait #1 and #2: 0.000000 sec
2. event: 'DFS lock handle'
time waited: 0.000150 sec
wait id: 41 p1: 'type|mode'=0x58420002
p2: 'id1'=0x0
p3: 'id2'=0x0
* time between wait #2 and #3: 0.000000 sec
3. event: 'ASM file metadata operation'
time waited: 0.013519 sec
wait id: 40 p1: 'msgop'=0x0
p2: 'locn'=0x4
}
and is blocked by
=> Oracle session identified by:
{
instance: 6 (+asm.+asm6)
os id: 18436
process id: 29, oracle@xxx.xxx.xxx.com (ARB0)
session id: 327
session serial #: 697
}
which is waiting for 'DFS lock handle' with wait info:
{
p1: 'type|mode'=0x43490005
p2: 'id1'=0x2b
p3: 'id2'=0x5
time in wait: 799 min 51 sec
timeout after: never
wait id: 3276323
blocking: 1 session
current sql: <none>
short stack: ksedsts()+461<-ksdxfstk()+32<-ksdxcb()+1876<-sspuser()+112<-__sighandler()<-semtimedop()+10<-skgpwwait()+160<-ksliwat()+1865<-kslwaitctx()+163<-kjusuc()+3619<-ksigeti()+3781<-ksbcic_int()+5092<-ksbcic()+12<-kffMapLock()+468<-kffRelocateMapLock()+394<-kffRelocateClientData()+3899<-kffRelocateSubmit()+43<-kffRelocate()+1642<-kfdaExecute()+3802<-kfdCompact()+2319<-kfdExecute()+1690<-kfgbRebalExecute()+874<-kfgbDriver()+1899<-ksbabs()+771<-kfgbRun()+838<-ksbrdp()+971<-opirip()+623<-opidrv()+603<-sou2o()+103<-opimai_re
wait history:
* time between current wait and wait #1: 0.000028 sec
1. event: 'DFS lock handle'
time waited: 0.000173 sec
wait id: 3276322 p1: 'type|mode'=0x41520005
p2: 'id1'=0x300e924
p3: 'id2'=0x533
* time between wait #1 and #2: 0.000049 sec
2. event: 'DFS lock handle'
time waited: 0.088700 sec
wait id: 3276321 p1: 'type|mode'=0x41520005
p2: 'id1'=0x300e924
p3: 'id2'=0x532
* time between wait #2 and #3: 0.000054 sec
3. event: 'enq: AD - allocate AU'
time waited: 0.000108 sec
wait id: 3276320 p1: 'name|mode'=0x41440006
p2: 'group and disk number'=0x3003c
p3: 'AU number'=0x3cb7
}
and is blocked by
=> Oracle session identified by:
{
instance: 1 (+asm.+asm1)
os id: 27356
process id: 33, oracle@xxx.xxx.xxx.com (ARB0)
session id: 587
session serial #: 5261
}
which is waiting for 'DFS lock handle' with wait info:
{
p1: 'type|mode'=0x43490005
p2: 'id1'=0x2b
p3: 'id2'=0x2
time in wait: 799 min 51 sec
timeout after: never
wait id: 11448589
blocking: 3 sessions
current sql: <none>
short stack: ksedsts()+461<-ksdxfstk()+32<-ksdxcb()+1876<-sspuser()+112<-__sighandler()<-semtimedop()+10<-skgpwwait()+160<-ksliwat()+1865<-kslwaitctx()+163<-kjusuc()+3619<-ksigeti()+3781<-ksbcic_int()+21066<-ksbcic()+12<-kffMapLock()+468<-kffRelocateMapLock()+394<-kffRelocateClientData()+3899<-kffRelocateSubmit()+43<-kffRelocate()+1642<-kfdaExecute()+3802<-kfdCompact()+2319<-kfdExecute()+1690<-kfgbRebalExecute()+874<-kfgbDriver()+1899<-ksbabs()+771<-kfgbRun()+838<-ksbrdp()+971<-opirip()+623<-opidrv()+603<-sou2o()+103<-opimai_r
wait history:
* time between current wait and wait #1: 0.000030 sec
1. event: 'DFS lock handle'
time waited: 0.000253 sec
wait id: 11448588 p1: 'type|mode'=0x43490005
p2: 'id1'=0x2b
p3: 'id2'=0x3
* time between wait #1 and #2: 0.000031 sec
2. event: 'DFS lock handle'
time waited: 0.000373 sec
wait id: 11448587 p1: 'type|mode'=0x43490005
p2: 'id1'=0x2b
p3: 'id2'=0x1
* time between wait #2 and #3: 0.000018 sec
3. event: 'DFS lock handle'
time waited: 0.000220 sec
wait id: 11448586 p1: 'type|mode'=0x41520005
p2: 'id1'=0x1000561
p3: 'id2'=0x14cb
}
and is blocked by
=> Oracle session identified by:
{
instance: 4 (+asm.+asm4)
os id: 25283
process id: 17, oracle@xxx.xxx.xxx.com (CKPT)
session id: 1106
session serial #: 1
}
which is waiting for 'rdbms ipc message' with wait info:
{
p1: 'timeout'=0xaa
time in wait: 0.873425 sec
heur. time in wait: 0.873779 sec
timeout after: 0.826575 sec
wait id: 110789577
blocking: 4 sessions
current sql: <none>
short stack: ksedsts()+461<-ksdxfstk()+32<-ksdxcb()+1876<-sspuser()+112<-__sighandler()<-semtimedop()+10<-skgpwwait()+160<-ksliwat()+1865<-kslwaitctx()+163<-kslwait()+141<-ksarcv()+207<-ksbabs()+330<-ksbrdp()+971<-opirip()+623<-opidrv()+603<-sou2o()+103<-opimai_real()+266<-ssthrdmain()+252<-main()+201<-__libc_start_main()+244<-_start()+36
wait history:
* time between current wait and wait #1: 0.000005 sec
1. event: 'rdbms ipc message'
time waited: 0.000103 sec
wait id: 110789576 p1: 'timeout'=0xaa
* time between wait #1 and #2: 0.000003 sec
2. event: 'rdbms ipc message'
time waited: 0.000004 sec
wait id: 110789575 p1: 'timeout'=0xaa
* time between wait #2 and #3: 0.000006 sec
3. event: 'rdbms ipc message'
time waited: 0.000224 sec
wait id: 110789574 p1: 'timeout'=0xaa
}
Chain 1 Signature: 'rdbms ipc message'<='DFS lock handle'<='DFS lock handle'<='DFS lock handle'
Chain 1 Signature Hash: 0xd945e807
-------------------------------------------------------------------------------
File_name :: +ASM4_diag_25258.trc
Changes
Cause
To view full details, sign in with your My Oracle Support account. |
|
Don't have a My Oracle Support account? Click to get started! |
In this Document
| Symptoms |
| Changes |
| Cause |
| Solution |
| References |