My Oracle Support Banner

ARC not shrinking due to zfs_arc_min overcapped causing performance degradation or hang (Doc ID 2131419.1)

Last updated on JULY 20, 2023

Applies to:

Solaris Operating System - Version 10 10/08 U6 and later
Information in this document applies to any platform.

Symptoms

The system suffers serious performance issues as it is hard swapping, however we see memory not being released from the Adaptive Replacement Cache (ARC).

We see the following in a crashdump.

crash file:
release: 5.11 (64-bit)
version: 11.3
hw_provider: Oracle Corporation
system type: ORCL,SPARC-T5-2 (SPARC-T5)
..
panic CPU: 0 (16 CPUs, 64G memory, 2 nodes)
panic string: Panic - Generated at user request

sanity checks: settings...
..
WARNING: needfree is 3331 pages
WARNING: 4 severe kstat errors (run "kstat xck")
WARNING: page_create() throttled (freemem < throttlefree)
WARNING: hard swapping (avefree < minfree)
NOTE: nswapped is 123
disks...done

 

Page               Summary        Pages Bytes   %Tot
----------------- ---------------- ---------------- ----
Kernel             750287            5.7G             9%
ZFS Metadata   111231           868.9M          1%
ZFS File Data   7625692          58.1G            91% <= !!!
Anon               1757               13.7M            0%
Exec and libs    177                1.3M              0%
Page cache      635                 4.9M              0%
Free (cachelist) 0                    0                   0%
Free (freelist)   32040             250.3M           0%
Total 8388608   64G

The largest memory usage is from the ZFS File Data

> ARC_mfu::print s_evictable.l_size
s_evictable.l_size = 0x1f78c0000

We have a lot waiting for eviction but nothing seem to be happening

> ::arc
size = 60733 MB
target size (c) = 64310 MB
target mru_size (p) = 4019 MB
c_min = 64310 MB <=!!
c_max = 64310 MB
buf_size = 214 MB
data_size = 60251 MB
other_size = 250 MB
rawdata_size = 0 MB
meta_used = 481 MB
meta_max = 481 MB
meta_limit = 0 MB
memory_throttle_count = 12255
arc_no_grow = 1
arc_tempreserve = 0 MB
mfu_size = 8058 MB
mru_size = 52135 MB

 

c_min (zfs_arc_min) is really high (basically close to zfs_arc_max) which is itself close to host physmem

> ::system
...
set zfs:zfs_arc_max=0x3e66666666 [0t268005959270] <= !!!!!!
set zfs:zfs_arc_min=0x1eb851eb85 [0t131941395333] <= !!!!!!

zfs_arc_min and zfs_arc_max have been set higher than the system memory

ZFS is then ajusting these 2 values as they are outside boundaries.

 

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!

In this Document
Symptoms
Cause
Solution
References


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.