ZFS ARC can shrink down without memory pressure result in slow performance (Doc ID 1404581.1)

Last updated on SEPTEMBER 12, 2016

Applies to:

Solaris Operating System - Version 11 11/11 to 11.1 [Release 11.0]
Information in this document applies to any platform.

Symptoms

The ZFS ARC (Adaptive Replacement Cache) is used for caching file system blocks. It has been noticed that the ARC can shrink too low without any sign of memory pressure and remain there on Solaris 11.

When this happens, L2ARC (if configured) is not fed any more data. That may cause slow application performance.

This issue can happen after ARC size increases to it's maximum limit (3/4 of physical memory by default). It then continues to drop too low and the flag arc_no_grow is set to TRUE and not changed back to FALSE.

You can check the size of ARC and  this flag by running:

# echo '::arc' | mdb -k | egrep '^(size|arc_no_grow)'

size = 1882 MB
arc_no_grow = 1


Another symptom is that the vmem arena shows zfs_file_data has a very large value but zfs_file_data_buf is very small.

# kstat -p | egrep 'zfs_file_data:mem_inuse|zfs_file_data:mem_total'

vmem:32:zfs_file_data:mem_inuse 16314793984
vmem:32:zfs_file_data:mem_total 17171480576

# kstat -p | egrep 'zfs_file_data_buf:mem_inuse|zfs_file_data_buf:mem_total'

kstat-p.out:vmem:33:zfs_file_data_buf:mem_inuse 12251136
kstat-p.out:vmem:33:zfs_file_data_buf:mem_total 12251136


Running mdb(1) ::memstat dcmd shows very little memory in the "ZFS File Data" row even with plenty of free memory available:

# echo ::memstat|mdb -k

Page Summary            Pages               MB %Tot
------------ ---------------- ---------------- ----
Kernel                 842243             3290 20%
ZFS File Data            3028               11  0%
Anon                    43638              170  1%
Exec and libs            3712               14  0%
Page cache               8573               33  0%
Free (cachelist)         5530               21  0%
Free (freelist)       3285288            12833 78%

Total                 4192012            16375
Physical              4192011            16375

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms