ASM Hang KFC LRU latch (Doc ID 1933081.1)

Last updated on JUNE 28, 2017

Applies to:

Oracle Database - Enterprise Edition - Version 11.1.0.7 to 11.2.0.1 [Release 11.1 to 11.2]
Information in this document applies to any platform.

Symptoms

ASM instance is getting hang on large cluster without any visible changes.

Resulting all instance getting hang .

LMS waiting for LRU latch ,

*** 2014-09-24 01:36:39.717
==============================
kjfmHBcheck: LMS0 (ospid: 12488) not moving for 80 sec (1411540599.1411540519)
 : waiting for latch 'KFC Hash Latch'.
 : waiting for event 'latch free' for 259 secs with wait_id 33014438.
==============================
===[ Waiter Latch State: 0x78f926b0 ]===
 (post info) last post received: 0 0 318
             last post received-location: kfc2.h LINE:842 ID:kfcDropLatchPost02
             last process to post me: 78f8a570 1 14
             last post sent: 0 0 123
             last post sent-location: kjm.h LINE:1134 ID:kjmsm: domain deletion sync with lmd
             last process posted by me: 78f91570 1 6
 (latch info) wait_event=0 bits=0
       Location from where call was made: kfc2.h LINE:678 ID:kfclNtIter:
       Context saved from call: 31
   waiting for 75754120 Child KFC Hash Latch level=2 child#=3
       Location from where latch is held: kfc2.h LINE:568 ID:kfcget:
       Context saved from call: 1
       state=busy [holder orapid=36] wlstate=free [value=0]
       waiters [orapid (seconds since: put on list, posted, alive check)]:
        4 (20339, 1411540599, 1)
        42 (20158, 1411540599, 20158)
        22 (20049, 1411540599, 20049)
        38 (19621, 1411540599, 19621)
        46 (18190, 1411540599, 18190)
        37 (17712, 1411540599, 17712)
       
        10 (260, 1411540599, 2)
        waiter count=37
 Process Group: DEFAULT, pseudo proc: 0x78486838
 O/S info: user: oracle, term: UNKNOWN, ospid: 14253
 OSD pid info: Unix process pid: 14253, image: oracle@ausmegaprddb11.us.dell.com (TNS V1-V3)
Short stack dump:
<-ksedsts()+315<-ksdxfstk()+32<-ksdxcb()+1764<-sspuser()+112<-0000003A03E0DE70<-semop()+7<-sskgpwwait()+980<-skgpwwait()+156<-kslges()+1240<-kslgetl()+462<-kfcGetBuffer()+497<-kfcReadBuffer()+6980<-kfcGet0()+21401<-kfcGet1Priv()+964<-kfcGetPriv()+213<-kffbSetLimits()+185<-kffbGetBlkPriv()+146<-kffdScanBegin()+93<-kfdDdScanBegin()+13<-kfgComputeGrpProp()+216<-kfgGrpEntryCb()+587<-kfgTableCb()+511<-kfgGrpTableCbInternal()+251<-kfgGrpTableCb1()+56<-qerfxFetch()+8704<-opifch2()+3570<-opifch()+64<-opiodr()+1220<-ttcpip()+1283<-opitsk()+1494<-opiino()+1026<-opiodr()+1220<-opidrv()+580<-sou2o()+90<-opimai_real()+145<-ssthrdmain()+177<-main()+215<-__libc_start_main()+244<-_start()+41
Dump of memory from 0x0000000078F73850 to 0x0000000078F73A58
078F73850 00000000 00000000 00000000 00000000  [................]
       Repeat 31 times
078F73A50 00000000 00000000                    [........]        
 Current Wait Stack:
  0: waiting for 'latch free'
     address=6003df70, number=1c6, tries=0
     wait_id=4 seq_num=5 snap_id=1
     wait times: snap=349 min 46 sec, exc=349 min 46 sec, total=349 min 46 sec
     wait times: max=infinite
     wait counts: calls=0 os=0
     in_wait=1 iflags=0x2520
 There is at least one session blocking this session.
 Dumping one blocker:
   inst: 11, sid: 210, ser: 1
 There is at least one session blocked by this session.
 Dumping one waiter:
   inst: 11, sid: 222, ser: 1
   wait event: 'latch free'
     p1: 'address'=0x75754120
     p2: 'number'=0x1c7
     p3: 'tries'=0x0
   row_wait_obj#: 4294967295, block#: 0, row#: 0, file# 0
   min_blocked_time: 20311 secs, waiter_cache_ver: 12474
 Wait State:
   auto_close=0 flags=0x20 boundary=(nil)/-1
 Session Wait History:
  0: waited for 'SQL*Net message from client'
     driver id=62657100, #bytes=1, =0
     wait_id=3 seq_num=4 snap_id=1
     wait times: snap=0.000177 sec, exc=0.000177 sec, total=0.000177 sec
     wait times: max=infinite
     wait counts: calls=0 os=0
     occurred after 0.000098 sec of elapsed time
  1: waited for 'SQL*Net message to client'
     driver id=62657100, #bytes=1, =0
     wait_id=2 seq_num=3 snap_id=1
     wait times: snap=0.000001 sec, exc=0.000001 sec, total=0.000001 sec
     wait times: max=infinite
     wait counts: calls=0 os=0
     occurred after 0.000372 sec of elapsed time

 
File_name :: +ASM11_lmon_12484.trc

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms