My Oracle Support Banner

ZFS Performance Degrades With Threads Held Up By kcage_create_throttle() (Doc ID 1528282.1)

Last updated on JULY 23, 2018

Applies to:

Solaris Operating System - Version 10 6/06 U2 to 11 11/11 [Release 10.0 to 11.0]
Information in this document applies to any platform.
***Checked for relevance on 09-Aug-2016***

Symptoms

ZFS IO performance can occasionally degrade severely.
If this issue is encountered then a threadlist taken from the system when performance is degraded will show several threads with a stack similar to the following:

000002a10e483ca0      600c978c6b0      301e4923418   2  99          1878448
  PC: cv_wait+0x38    CMD: zpool-datapool
  stack pointer for thread 2a10e483ca0: 2a10e482121
  [ 000002a10e482121 cv_wait+0x38() ]
    kcage_create_throttle+0x188()
    page_create_va+0xe8()
    segkmem_page_create+0x84()
    segkmem_xalloc+0xb0()
    segkmem_alloc_vn+0xc0()
    vmem_xalloc+0x5ac()
    vmem_alloc+0x1d4()
    kmem_slab_create+0x44()
    kmem_slab_alloc+0x30()
    kmem_cache_alloc+0x2dc()
    arc_buf_alloc+0x14()
    arc_write_done+0x64()
    zio_done+0x448()
    zio_execute+0xb4()
    zio_done+0x4a4()
    zio_execute+0xb4()
    taskq_thread+0x3b8()
    thread_start+4()

000003001b064140      600c23a9260      30232d0e890   1  60          1878448
  PC: cv_wait+0x38    CMD: app
  stack pointer for thread 3001b064140: 2a127179ad1
  [ 000002a127179ad1 cv_wait+0x38() ]
    kcage_create_throttle+0x188()
    page_create_va+0xe8()
    segkmem_page_create+0x84()
    segkmem_xalloc+0xb0()
    segkmem_alloc_vn+0xc0()
    vmem_xalloc+0x5ac()
    vmem_alloc+0x1d4()
    kmem_slab_create+0x44()
    kmem_slab_alloc+0x30()
    kmem_cache_alloc+0x2dc()
    dbuf_create+0x1c()
    dbuf_create_bonus+0x14()
    dmu_bonus_hold+0x78()
    zfs_zget+0x30()
    zfs_dirent_lock+0x488()
    zfs_dirlook+0x1d0()
    zfs_lookup+0x3bc()
    zfs_shim_lookup+0x30()
    fop_lookup+0x28()
    lookuppnvp+0x354()
    lookuppnat+0x120()
    lookupnameat+0xa8()
    vn_openat+0x16c()
    copen+0x260()
    syscall_trap+0xac()

 
The fingerprint is that the stack contains kcage_create_throttle(), and kmem_cache_alloc() is called from ZFS related functions.

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


In this Document
Symptoms
Cause
Solution
References


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.