ASM Hang:: ASM rebalance slave process (ARBn) blocked by ASM CKPT (Doc ID 1565248.1)

Last updated on OCTOBER 31, 2016

Applies to:

Oracle Database - Enterprise Edition - Version 11.2.0.3 and later
Information in this document applies to any platform.

Symptoms

The rebalance operation for DATA and RECO are running on nodes 1 and 6. 

Rebalance process is hanged.Database is up and running ,properly operation.

 

Pstack of asm_rbal process looks like below,

#4  0x0000000007f823db in kslwaitctx ()
#5  0x000000000425069b in kjusuc ()
#6  0x00000000063343eb in ksigeti ()
#7  0x000000000215cea6 in ksbcic_int ()
#8  0x0000000002157c56 in ksbcic ()
#9  0x0000000003d4571c in kffMapLock ()
#10 0x0000000003d45496 in kffRelocateMapLock ()
#11 0x0000000003d4f605 in kffRelocateClientData ()
#12 0x0000000003d72a73 in kffRelocateSubmit ()
#13 0x0000000003d73174 in kffRelocate ()
#14 0x0000000003cf4026 in kfdaExecute ()
#15 0x0000000003cc3d29 in kfdCompact ()
#16 0x0000000003cc4cc2 in kfdExecute ()
#17 0x0000000003dc4dda in kfgbRebalExecute ()
#18 0x0000000003db24f3 in kfgbDriver ()
#19 0x0000000002160345 in ksbabs ()
#20 0x0000000003dc897a in kfgbRun ()

 

 Oracle session identified by:
   {
               instance: 4 (+asm.+asm4)
                  os id: 2321
             process id: 51, oracle@xxx.xxx.xxx.com (TNS V1-V3)
             session id: 198
       session serial #: 26865
   }
   is waiting for 'DFS lock handle' with wait info:
   {
                     p1: 'type|mode'=0x43490005
                     p2: 'id1'=0x2b
                     p3: 'id2'=0x5
           time in wait: 37 min 40 sec
          timeout after: never
                wait id: 42
               blocking: 0 sessions
            current sql: alter diskgroup RECO_USP1 rebalance power 32
            short stack: ksedsts()+461<-ksdxfstk()+32<-ksdxcb()+1876<-sspuser()+112<-__sighandler()<-semtimedop()+10<-skgpwwait()+160<-ksliwat()+1865<-kslwaitctx()+163<-kjusuc()+3619<-ksigeti()+3781<-ksbcic_int()+5092<-ksbcicr()+207<-kfnmGroupBlockGlobal()+2216<-kfnmGroupBlockPriv()+282<-kfgFinalize()+339<-kfxdrvAlter()+3265<-kfxdrvEntry()+1306<-opiexe()+20165<-opiosq0()+3870<-kpooprx()+274<-kpoal8()+800<-opiodr()+916<-ttcpip()+2242<-opitsk()+1673<-opiino()+966<-opiodr()+916<-opidrv()+570<-sou2o()+103<-opimai_real()+133<-ssthrdmain()+252<
           wait history:
             * time between current wait and wait #1: 0.000000 sec
             1.       event: 'ASM file metadata operation'
                time waited: 0.000038 sec (last interval)
                time waited: 0.013707 sec (total)
                    wait id: 40              p1: 'msgop'=0x0
                                             p2: 'locn'=0x4
             * time between wait #1 and #2: 0.000000 sec
             2.       event: 'DFS lock handle'
                time waited: 0.000150 sec
                    wait id: 41              p1: 'type|mode'=0x58420002
                                             p2: 'id1'=0x0
                                             p3: 'id2'=0x0
             * time between wait #2 and #3: 0.000000 sec
             3.       event: 'ASM file metadata operation'
                time waited: 0.013519 sec
                    wait id: 40              p1: 'msgop'=0x0
                                             p2: 'locn'=0x4
   }
   and is blocked by
=> Oracle session identified by:
   {
               instance: 6 (+asm.+asm6)
                  os id: 18436
             process id: 29, oracle@xxx.xxx.xxx.com (ARB0)
             session id: 327
       session serial #: 697
   }
   which is waiting for 'DFS lock handle' with wait info:
   {
                     p1: 'type|mode'=0x43490005
                     p2: 'id1'=0x2b
                     p3: 'id2'=0x5
           time in wait: 799 min 51 sec
          timeout after: never
                wait id: 3276323
               blocking: 1 session
            current sql: <none>
            short stack: ksedsts()+461<-ksdxfstk()+32<-ksdxcb()+1876<-sspuser()+112<-__sighandler()<-semtimedop()+10<-skgpwwait()+160<-ksliwat()+1865<-kslwaitctx()+163<-kjusuc()+3619<-ksigeti()+3781<-ksbcic_int()+5092<-ksbcic()+12<-kffMapLock()+468<-kffRelocateMapLock()+394<-kffRelocateClientData()+3899<-kffRelocateSubmit()+43<-kffRelocate()+1642<-kfdaExecute()+3802<-kfdCompact()+2319<-kfdExecute()+1690<-kfgbRebalExecute()+874<-kfgbDriver()+1899<-ksbabs()+771<-kfgbRun()+838<-ksbrdp()+971<-opirip()+623<-opidrv()+603<-sou2o()+103<-opimai_re
           wait history:
             * time between current wait and wait #1: 0.000028 sec
             1.       event: 'DFS lock handle'
                time waited: 0.000173 sec
                    wait id: 3276322         p1: 'type|mode'=0x41520005
                                             p2: 'id1'=0x300e924
                                             p3: 'id2'=0x533
             * time between wait #1 and #2: 0.000049 sec
             2.       event: 'DFS lock handle'
                time waited: 0.088700 sec
                    wait id: 3276321         p1: 'type|mode'=0x41520005
                                             p2: 'id1'=0x300e924
                                             p3: 'id2'=0x532
             * time between wait #2 and #3: 0.000054 sec
             3.       event: 'enq: AD - allocate AU'
                time waited: 0.000108 sec
                    wait id: 3276320         p1: 'name|mode'=0x41440006
                                             p2: 'group and disk number'=0x3003c
                                             p3: 'AU number'=0x3cb7
   }
   and is blocked by
=> Oracle session identified by:
   {
               instance: 1 (+asm.+asm1)
                  os id: 27356
             process id: 33, oracle@xxx.xxx.xxx.com (ARB0)
             session id: 587
       session serial #: 5261
   }
   which is waiting for 'DFS lock handle' with wait info:
   {
                     p1: 'type|mode'=0x43490005
                     p2: 'id1'=0x2b
                     p3: 'id2'=0x2
           time in wait: 799 min 51 sec
          timeout after: never
                wait id: 11448589
               blocking: 3 sessions
            current sql: <none>
            short stack: ksedsts()+461<-ksdxfstk()+32<-ksdxcb()+1876<-sspuser()+112<-__sighandler()<-semtimedop()+10<-skgpwwait()+160<-ksliwat()+1865<-kslwaitctx()+163<-kjusuc()+3619<-ksigeti()+3781<-ksbcic_int()+21066<-ksbcic()+12<-kffMapLock()+468<-kffRelocateMapLock()+394<-kffRelocateClientData()+3899<-kffRelocateSubmit()+43<-kffRelocate()+1642<-kfdaExecute()+3802<-kfdCompact()+2319<-kfdExecute()+1690<-kfgbRebalExecute()+874<-kfgbDriver()+1899<-ksbabs()+771<-kfgbRun()+838<-ksbrdp()+971<-opirip()+623<-opidrv()+603<-sou2o()+103<-opimai_r
           wait history:
             * time between current wait and wait #1: 0.000030 sec
             1.       event: 'DFS lock handle'
                time waited: 0.000253 sec
                    wait id: 11448588        p1: 'type|mode'=0x43490005
                                             p2: 'id1'=0x2b
                                             p3: 'id2'=0x3
             * time between wait #1 and #2: 0.000031 sec
             2.       event: 'DFS lock handle'
                time waited: 0.000373 sec
                    wait id: 11448587        p1: 'type|mode'=0x43490005
                                             p2: 'id1'=0x2b
                                             p3: 'id2'=0x1
             * time between wait #2 and #3: 0.000018 sec
             3.       event: 'DFS lock handle'
                time waited: 0.000220 sec
                    wait id: 11448586        p1: 'type|mode'=0x41520005
                                             p2: 'id1'=0x1000561
                                             p3: 'id2'=0x14cb
   }
   and is blocked by
=> Oracle session identified by:
   {
               instance: 4 (+asm.+asm4)
                  os id: 25283
             process id: 17, oracle@xxx.xxx.xxx.com (CKPT)
             session id: 1106
       session serial #: 1
   }
   which is waiting for 'rdbms ipc message' with wait info:
   {
                     p1: 'timeout'=0xaa
           time in wait: 0.873425 sec
     heur. time in wait: 0.873779 sec
          timeout after: 0.826575 sec
                wait id: 110789577
               blocking: 4 sessions
            current sql: <none>
            short stack: ksedsts()+461<-ksdxfstk()+32<-ksdxcb()+1876<-sspuser()+112<-__sighandler()<-semtimedop()+10<-skgpwwait()+160<-ksliwat()+1865<-kslwaitctx()+163<-kslwait()+141<-ksarcv()+207<-ksbabs()+330<-ksbrdp()+971<-opirip()+623<-opidrv()+603<-sou2o()+103<-opimai_real()+266<-ssthrdmain()+252<-main()+201<-__libc_start_main()+244<-_start()+36
           wait history:
             * time between current wait and wait #1: 0.000005 sec
             1.       event: 'rdbms ipc message'
                time waited: 0.000103 sec
                    wait id: 110789576       p1: 'timeout'=0xaa
             * time between wait #1 and #2: 0.000003 sec
             2.       event: 'rdbms ipc message'
                time waited: 0.000004 sec
                    wait id: 110789575       p1: 'timeout'=0xaa
             * time between wait #2 and #3: 0.000006 sec
             3.       event: 'rdbms ipc message'
                time waited: 0.000224 sec
                    wait id: 110789574       p1: 'timeout'=0xaa
   }

Chain 1 Signature: 'rdbms ipc message'<='DFS lock handle'<='DFS lock handle'<='DFS lock handle'
Chain 1 Signature Hash: 0xd945e807
-------------------------------------------------------------------------------


File_name :: +ASM4_diag_25258.trc

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms