Large page cache (cachelist) is preventing ZFS ARC from growing affecting IO throughput

(Doc ID 2071109.1)

Last updated on JULY 29, 2016

Applies to:

Solaris Operating System - Version 11.2 and later
Information in this document applies to any platform.
Resembles BUG 19576400 - Kernel Cage allocations failing with large cache list

Symptoms

ZFS/STMF IO Speeds Are Very Slow After Updating Solaris.  
The affected configuration involves zvols mapped to iSCSI LUNs through STMF(COMSTAR).

1.  IO throughput slows dramatically.

2.  The size of ZFS ARC is small, approaching minimum.  arc_no_grow will also be enabled.  The current arc_c=arc_min and arp_p tell that we should shrink even more.

 # echo "::arc" | mdb -k
 p                         =           31 MB
 c                         =          127 MB   <=
 c_min                     =          127 MB   <=
 c_max                     =        31702 MB
 size                      =          155 MB
 memory_throttle_count     =            0
 arc_no_grow               =            1      <=


 Only relevant entries shown.

 
3.  The system shows significant amount of free memory.  Memory statistics (memstat) are showing a large portion freemem in cachelist.

  # echo "::memstat" | mdb -k
 Page Summary                 Pages             Bytes  %Tot
 ----------------- ----------------  ----------------  ----
 Kernel                     1091600              4.1G   13%
 Defdump prealloc            190078            742.4M    2%
 Guest                            0                 0    0%
 ZFS Metadata                 22007             85.9M    0%
 ZFS File Data                19093             74.5M    0%
 Anon                         21280             83.1M    0%
 Exec and libs                  855              3.3M    0%
 Page cache                  647823              2.4G    8%
 In temporary use                 1                4k    0%
 Free (cachelist)           6349938             24.2G   76%   <=
 Free (freelist)              39332            153.6M    0%
 Total                      8382007             31.9G


Looking at tile

 # echo "::tile -v" | mdb -k
             TILE M   ID   MG MN PRI K T   TOTAL    FREE     OOB  CACHED  BORROW
 ffffe30600000000      0    0  0   0 g k    3.9g  139.9m  133.8m    1.6g    2.0g
 ffffe30600007c40      1    0  0   - g k      4g       -       -    3.5g    3.8g
 ffffe3060000f880      2    0  0   - g k      4g    2.9m    2.5m    3.4g    3.8g
 ffffe306000174c0      3    0  0   - g k      4g    1.9m    1.5m    3.5g    3.8g
 ffffe3060001f100      4    0  0   - g k      4g    2.2m    1.6m    3.5g    3.8g
 ffffe30600026d40      5    0  0   - g k      4g    1.3m    1.2m    3.5g    3.8g
 ffffe3060002e980      6    0  0   - g k      4g    940k    844k    3.2g    3.5g
 ffffe306000365c0      7    0  0   - g k    3.9g       -       -    1.6g    1.7g
                                total k   31.9g  149.3m  141.5m   24.1g   26.7g

Everything is caged

 # echo "::predict" | mdb -k  
               BP MN KU  SZC     STATE    AGE   NEEDED
 ffffc1c0089e6000  0  u   4k  Disabled 554363        -
 ffffc1c008a00000  0  u   2m  Disabled 936066        -
 ffffc1c008a12000  0  u   1g  Inbounds 3598996       -
 ffffc1c008400000  0  k   4k   Surplus      5        -
 ffffc1c0084ee000  0  k   2m  Disabled 772189        -
 ffffc1c0089d4000  0  k   1g  Inbounds 3607468       -

 # echo "::breadline" | mdb -k
        BREADLINE MN T  SZC APD  Q'D   NEEDED SOUPLINE  SATISFIED  ABANDONED
 ffffe3076a000000  0 u   4k  PD    -        -        -      945e4      a285a
 ffffe3076a06c000  0 u   2m   D    -        -        -         97        605
 ffffe3076a0d8000  0 u   1g        -        -        -          -          -
 ffffe3076a000180  0 k   4k        -        -        -    da9fa8f      11fbd
 ffffe3076a06c180  0 k   2m   D    -        -        -        130         1a
 ffffe3076a0d8180  0 k   1g        -        -        -          -          -

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms