Memory exhaustion leading to system hang on Solaris 10u11 (or kernel 147147-26) with ZFS (Doc ID 1573006.1)

Last updated on AUGUST 01, 2016

Applies to:

Solaris Operating System - Version 10 1/13 U11 and later
Information in this document applies to any platform.

Symptoms

The system hangs due to memory exhaustion, the hang may be preceded by messages indicating there's a shortage of memory such as

  "error Couldn't fork: Resource temporarily unavailable"

Review of a forced system crash dump will likely show a large portion of the systems memory used by the kernel and threads stuck waiting on memory which may include threads with stacks such as this

 genunix:cv_wait+0x38()
 genunix:delay_common+0x80()
 genunix:delay+0x88()
 unix:page_resv+0x78()
 unix:segkmem_xalloc+0x74()
 unix:segkmem_alloc_vn+0xc0()
 genunix:vmem_xalloc+0x5c8()
 genunix:vmem_alloc+0x1d4()
 genunix:kmem_slab_create+0x44()
 genunix:kmem_slab_alloc+0x30()
 genunix:kmem_cache_alloc+0x2dc()
 ..

 

Changes

The issue can affect SPARC and x64 systems running Solaris 10u11 using ZFS. 

  SPARC systems with kernel patch 147147-26 or later and without ZFS patch 150125-03. 

  x64 systems with kernel patch 147148-26 or later and without ZFS patch 149637-04.

The issue may be triggered by a backup on small memory systems (2gb), on larger memory systems the trigger may be an IO load particularly one that sequentially accesses datasets with different recordsizes.

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms