ZFS Performance Degrades With Threads Held Up By kcage_create_throttle()
(Doc ID 1528282.1)
Last updated on JULY 23, 2018
Applies to:
Solaris Operating System - Version 10 6/06 U2 to 11 11/11 [Release 10.0 to 11.0]Information in this document applies to any platform.
***Checked for relevance on 09-Aug-2016***
Symptoms
ZFS IO performance can occasionally degrade severely.
If this issue is encountered then a threadlist taken from the system when performance is degraded will show several threads with a stack similar to the following:
000002a10e483ca0 600c978c6b0 301e4923418 2 99 1878448
PC: cv_wait+0x38 CMD: zpool-datapool
stack pointer for thread 2a10e483ca0: 2a10e482121
[ 000002a10e482121 cv_wait+0x38() ]
kcage_create_throttle+0x188()
page_create_va+0xe8()
segkmem_page_create+0x84()
segkmem_xalloc+0xb0()
segkmem_alloc_vn+0xc0()
vmem_xalloc+0x5ac()
vmem_alloc+0x1d4()
kmem_slab_create+0x44()
kmem_slab_alloc+0x30()
kmem_cache_alloc+0x2dc()
arc_buf_alloc+0x14()
arc_write_done+0x64()
zio_done+0x448()
zio_execute+0xb4()
zio_done+0x4a4()
zio_execute+0xb4()
taskq_thread+0x3b8()
thread_start+4()
000003001b064140 600c23a9260 30232d0e890 1 60 1878448
PC: cv_wait+0x38 CMD: app
stack pointer for thread 3001b064140: 2a127179ad1
[ 000002a127179ad1 cv_wait+0x38() ]
kcage_create_throttle+0x188()
page_create_va+0xe8()
segkmem_page_create+0x84()
segkmem_xalloc+0xb0()
segkmem_alloc_vn+0xc0()
vmem_xalloc+0x5ac()
vmem_alloc+0x1d4()
kmem_slab_create+0x44()
kmem_slab_alloc+0x30()
kmem_cache_alloc+0x2dc()
dbuf_create+0x1c()
dbuf_create_bonus+0x14()
dmu_bonus_hold+0x78()
zfs_zget+0x30()
zfs_dirent_lock+0x488()
zfs_dirlook+0x1d0()
zfs_lookup+0x3bc()
zfs_shim_lookup+0x30()
fop_lookup+0x28()
lookuppnvp+0x354()
lookuppnat+0x120()
lookupnameat+0xa8()
vn_openat+0x16c()
copen+0x260()
syscall_trap+0xac()
PC: cv_wait+0x38 CMD: zpool-datapool
stack pointer for thread 2a10e483ca0: 2a10e482121
[ 000002a10e482121 cv_wait+0x38() ]
kcage_create_throttle+0x188()
page_create_va+0xe8()
segkmem_page_create+0x84()
segkmem_xalloc+0xb0()
segkmem_alloc_vn+0xc0()
vmem_xalloc+0x5ac()
vmem_alloc+0x1d4()
kmem_slab_create+0x44()
kmem_slab_alloc+0x30()
kmem_cache_alloc+0x2dc()
arc_buf_alloc+0x14()
arc_write_done+0x64()
zio_done+0x448()
zio_execute+0xb4()
zio_done+0x4a4()
zio_execute+0xb4()
taskq_thread+0x3b8()
thread_start+4()
000003001b064140 600c23a9260 30232d0e890 1 60 1878448
PC: cv_wait+0x38 CMD: app
stack pointer for thread 3001b064140: 2a127179ad1
[ 000002a127179ad1 cv_wait+0x38() ]
kcage_create_throttle+0x188()
page_create_va+0xe8()
segkmem_page_create+0x84()
segkmem_xalloc+0xb0()
segkmem_alloc_vn+0xc0()
vmem_xalloc+0x5ac()
vmem_alloc+0x1d4()
kmem_slab_create+0x44()
kmem_slab_alloc+0x30()
kmem_cache_alloc+0x2dc()
dbuf_create+0x1c()
dbuf_create_bonus+0x14()
dmu_bonus_hold+0x78()
zfs_zget+0x30()
zfs_dirent_lock+0x488()
zfs_dirlook+0x1d0()
zfs_lookup+0x3bc()
zfs_shim_lookup+0x30()
fop_lookup+0x28()
lookuppnvp+0x354()
lookuppnat+0x120()
lookupnameat+0xa8()
vn_openat+0x16c()
copen+0x260()
syscall_trap+0xac()
The fingerprint is that the stack contains kcage_create_throttle(), and kmem_cache_alloc() is called from ZFS related functions.
Cause
To view full details, sign in with your My Oracle Support account. |
|
Don't have a My Oracle Support account? Click to get started! |
In this Document
| Symptoms |
| Cause |
| Solution |
| References |