Threads blocked, waiting for kflt_create_throttle (Doc ID 1681689.1)

Last updated on JULY 29, 2016

Applies to:

Solaris Operating System - Version 10 1/13 U11 to 10 1/13 U11 [Release 10.0]
Oracle Solaris on x86-64 (64-bit)
This only happens on Solaris 10 Update 11

Symptoms

The system was hung.  In this case the box was running VCS.  When GAB no longer got a heartbeat it panicked the system.

This is due to:

Bug 15858413 - relatively idle system hung from rcp with freemem available

release:        5.10 (64-bit)
version:        Generic_148889-01
panic string:   GAB: Port h halting system due to client process failure at [14:1105]


-- What is going on in the system:

CAT(vmcore.0/10X)> thread summary
       reference clock = panic_lbolt: 0x5f430a, panic_hrtime: 0x38e9b54aa855
 187   threads ran since 1 second before current tick (134 user, 53 kernel)
 355   threads ran since 1 minute before current tick (254 user, 101 kernel)

   2   TS_RUN threads (1 user, 1 kernel)
  44   TS_STOPPED threads (43 user, 1 kernel)
   0   TS_FREE threads
   0   !TS_LOAD (swapped) threads
2191*  TS_ON_SWAPQ (candidates for swapping) threads (1703 user, 488 kernel)

   9*  threads trying to get a mutex (7 user, 2 kernel)        <------- here is the issue, threads have been blocked for almost 40 minutes
         longest sleeping 39 minutes 52.11 seconds earlier
   3*  threads trying to get an rwlock (3 user, 0 kernel)
         longest sleeping 6 minutes 59.16 seconds earlier
2039   threads waiting for a condition variable (1582 user, 457 kernel)
   0   threads sleeping on a semaphore
  26   threads sleeping on a user-level sobj (26 user, 0 kernel)
   0   threads waiting for a UPI mutex
  41   threads sleeping on a shuttle (door) (41 user, 0 kernel)
   2*  stopped threads holding locks (2 user, 0 kernel)

   0   threads in biowait()
   2*  procs with SIGKILL posted (see "tlist killed")
   3*  threads with procs with SIGKILL posted (3 user, 0 kernel)
   4*  threads in vmem_xalloc() (4 user, 0 kernel)

   1   threads in dispatch queues (0 user, 1 kernel)
   1*  threads in dispq of CPU running idle thread (0 user, 1 kernel)
  12*  threads with > 90% CPU (0 user, 12 kernel)

1 reader vnode found
count       vnode        vnode.v_path
===== ================== ============
  57            0x20000 (0x20000)

1 writer vnode found
count       vnode        vnode.v_path
===== ================== ============
  20            0x20000 (0x20000)

2191   total threads in allthreads list (1703 user, 488 kernel)
  16   thread_reapcnt
   0   lwp_reapcnt
2315   nthread


-- What threads are stuck in kernel locks:

CAT(vmcore.0/10X)> tlist -s idle sobj locks
 thread             pri  pctcpu              idle   PID              wchan command
 0xfffffe98ce205800  59   0.019             2.17s 22435 0xfffffe98cfb314d0 /usr/sap/IR1/SCS01/exe/sapstartsrv pf=/usr/sap/IR1/SYS/profile/IR1_SCS01_sapir1
 0xffffffffde5223a0  59   0.000          3m23.00s  2373 0xfffffe98d26c51f0 /usr/sbin/syslogd
 0xfffffe9937b3ac40  60   0.000          6m59.11s  4092 0xfffffffffbca4020 /usr/sap/hostctrl/exe/saposcol -s pf=/usr/sap/hostctrl/exe/host_profile
 0xfffffe98ce0a0000  60   0.000          6m59.16s  6495 0xfffffe99312cf678 /usr/sap/hostctrl/exe/saposcol -l -w60 pf=/usr/sap/hostctrl/exe/host_profile
 0xfffffe98d33f7ba0  59   0.000          7m17.94s  3987 0xfffffffffbca4020 sh -c /usr/bin/uname -r 1> /tmp/release
 0xfffffe98d5f0e3c0  60   0.000          7m18.02s  5063 0xfffffffffbca4020 /opt/VRTSagents/ha/bin/Netlsnr/NetlsnrAgent -type Netlsnr -agdir /opt/VRTSagent
 0xfffffe98ce0fb040  59   0.000         16m18.52s  4499 0xfffffffffbca4020 ora_pmon_IR1
 0xffffffffde51d440  59   0.000         38m58.15s 22496 0xfffffffffbca4020 /usr/sap/IR1/jre1.6.0_14/bin/java -Djava.endorsed.dirs=/usr/sap/IR1/mft/ixsclie
 0xfffffe98ce1ac740  59   0.000         38m58.15s 22496 0xfffffffffbca4020 /usr/sap/IR1/jre1.6.0_14/bin/java -Djava.endorsed.dirs=/usr/sap/IR1/mft/ixsclie
 0xfffffe800097cc40   0   0.000         39m52.04s     4 0xfffffffffbca4020 kmem_task
 0xfffffe98ce24b760  59   0.000         39m52.11s 15700 0xfffffffffbca4020 /opt/hps/inventory/bin/HPS_SCANNER_solaris-x86 -p:/var/log/hps/inventory -cfg:/
 0xfffffe800309ac40  60   0.000         39m52.11s     0 0xfffffffffbca4020 sched

 12 matching threads found
   with sobj lock

top mutex/rwlock owners:
count   thread
   9   0xfffffe992bae5c20  state: slp   wchan: 0xfffffffffbc0d350  sobj: condition var (from unix:kflt_create_throttle+0x13b)   <----------- here is our cluprit
   2   read-locked rwlocks (count:rwlock 1:0xfffffe99312cf678 1:0xfffffe98cfb314d0)
   1   0xffffffffbf0007c0  state: slp   wchan: 0xffffffffcc86cf2a  sobj: condition var (from ufs:top_begin_async+0x78)

-- Looking at that thread holding the mutex:


CAT(vmcore.0/10X)> thread 0xfffffe992bae5c20
<...>
unix:_resume_from_idle+0xfb resume_return()
unix:swtch - frame recycled
void genunix:cv_wait+0x68((kcondvar_t *), (kmutex_t *))
int unix:kflt_create_throttle+0x13b((pgcnt_t), (int))     <----------------------------------------------------------- here is the routine
page_t *unix:page_create_va+0x4e8((vnode_t *), (u_offset_t), (size_t), (uint_t), (struct seg *), (caddr_t))
page_t *unix:segkmem_page_create+0x7d((void *), (size_t), (int), (void *))
void *unix:segkmem_xalloc+0x60((vmem_t *), (void *), (size_t), (int), (uint_t), (page_t **()), (void *))
void *unix:segkmem_alloc_vn+0x8a((vmem_t *), (size_t), (int), (struct vnode *))
void *unix:segkmem_alloc+0x10((vmem_t *), (size_t), (int))
void *genunix:vmem_xalloc+0x322((vmem_t *), (size_t), (size_t), (size_t), (size_t), (void *), (void *), (int))
void *genunix:vmem_alloc+0x155((vmem_t *), (size_t), (int))
kmem_slab_t *genunix:kmem_slab_create+0x77((kmem_cache_t *), (int))
void *genunix:kmem_slab_alloc+0x52((kmem_cache_t *), (int))
void *genunix:kmem_cache_alloc+0x1f7((kmem_cache_t *), (int))
<...>

 

Changes

 This only happens on Solaris 10 Update 11

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms