Large page cache (cachelist) is preventing ZFS ARC from growing affecting IO throughput
(Doc ID 2071109.1)
Last updated on JULY 07, 2023
Applies to:
Solaris Operating System - Version 11.2 to 11.3 [Release 11.0]Information in this document applies to any platform.
Resembles BUG 19576400 - Kernel Cage allocations failing with large cache list
Symptoms
ZFS/STMF IO Speeds Are Very Slow After Updating Solaris.
The affected configuration involves zvols mapped to iSCSI LUNs through STMF(COMSTAR).
1. IO throughput slows dramatically.
2. The size of ZFS ARC is small, approaching minimum. arc_no_grow will also be enabled. The current arc_c=arc_min and arp_p tell that we should shrink even more.
# echo "::arc" | mdb -k
p = 31 MB
c = 127 MB <=
c_min = 127 MB <=
c_max = 31702 MB
size = 155 MB
memory_throttle_count = 0
arc_no_grow = 1 <=
Only relevant entries shown.
p = 31 MB
c = 127 MB <=
c_min = 127 MB <=
c_max = 31702 MB
size = 155 MB
memory_throttle_count = 0
arc_no_grow = 1 <=
Only relevant entries shown.
3. The system shows significant amount of free memory. Memory statistics (memstat) are showing a large portion freemem in cachelist.
# echo "::memstat" | mdb -k
Page Summary Pages Bytes %Tot
----------------- ---------------- ---------------- ----
Kernel 1091600 4.1G 13%
Defdump prealloc 190078 742.4M 2%
Guest 0 0 0%
ZFS Metadata 22007 85.9M 0%
ZFS File Data 19093 74.5M 0%
Anon 21280 83.1M 0%
Exec and libs 855 3.3M 0%
Page cache 647823 2.4G 8%
In temporary use 1 4k 0%
Free (cachelist) 6349938 24.2G 76% <=
Free (freelist) 39332 153.6M 0%
Total 8382007 31.9G
Looking at tile
# echo "::tile -v" | mdb -k
TILE M ID MG MN PRI K T TOTAL FREE OOB CACHED BORROW
ffffe30600000000 0 0 0 0 g k 3.9g 139.9m 133.8m 1.6g 2.0g
ffffe30600007c40 1 0 0 - g k 4g - - 3.5g 3.8g
ffffe3060000f880 2 0 0 - g k 4g 2.9m 2.5m 3.4g 3.8g
ffffe306000174c0 3 0 0 - g k 4g 1.9m 1.5m 3.5g 3.8g
ffffe3060001f100 4 0 0 - g k 4g 2.2m 1.6m 3.5g 3.8g
ffffe30600026d40 5 0 0 - g k 4g 1.3m 1.2m 3.5g 3.8g
ffffe3060002e980 6 0 0 - g k 4g 940k 844k 3.2g 3.5g
ffffe306000365c0 7 0 0 - g k 3.9g - - 1.6g 1.7g
total k 31.9g 149.3m 141.5m 24.1g 26.7g
Everything is caged
# echo "::predict" | mdb -k
BP MN KU SZC STATE AGE NEEDED
ffffc1c0089e6000 0 u 4k Disabled 554363 -
ffffc1c008a00000 0 u 2m Disabled 936066 -
ffffc1c008a12000 0 u 1g Inbounds 3598996 -
ffffc1c008400000 0 k 4k Surplus 5 -
ffffc1c0084ee000 0 k 2m Disabled 772189 -
ffffc1c0089d4000 0 k 1g Inbounds 3607468 -
# echo "::breadline" | mdb -k
BREADLINE MN T SZC APD Q'D NEEDED SOUPLINE SATISFIED ABANDONED
ffffe3076a000000 0 u 4k PD - - - 945e4 a285a
ffffe3076a06c000 0 u 2m D - - - 97 605
ffffe3076a0d8000 0 u 1g - - - - -
ffffe3076a000180 0 k 4k - - - da9fa8f 11fbd
ffffe3076a06c180 0 k 2m D - - - 130 1a
ffffe3076a0d8180 0 k 1g - - - - -
Page Summary Pages Bytes %Tot
----------------- ---------------- ---------------- ----
Kernel 1091600 4.1G 13%
Defdump prealloc 190078 742.4M 2%
Guest 0 0 0%
ZFS Metadata 22007 85.9M 0%
ZFS File Data 19093 74.5M 0%
Anon 21280 83.1M 0%
Exec and libs 855 3.3M 0%
Page cache 647823 2.4G 8%
In temporary use 1 4k 0%
Free (cachelist) 6349938 24.2G 76% <=
Free (freelist) 39332 153.6M 0%
Total 8382007 31.9G
Looking at tile
# echo "::tile -v" | mdb -k
TILE M ID MG MN PRI K T TOTAL FREE OOB CACHED BORROW
ffffe30600000000 0 0 0 0 g k 3.9g 139.9m 133.8m 1.6g 2.0g
ffffe30600007c40 1 0 0 - g k 4g - - 3.5g 3.8g
ffffe3060000f880 2 0 0 - g k 4g 2.9m 2.5m 3.4g 3.8g
ffffe306000174c0 3 0 0 - g k 4g 1.9m 1.5m 3.5g 3.8g
ffffe3060001f100 4 0 0 - g k 4g 2.2m 1.6m 3.5g 3.8g
ffffe30600026d40 5 0 0 - g k 4g 1.3m 1.2m 3.5g 3.8g
ffffe3060002e980 6 0 0 - g k 4g 940k 844k 3.2g 3.5g
ffffe306000365c0 7 0 0 - g k 3.9g - - 1.6g 1.7g
total k 31.9g 149.3m 141.5m 24.1g 26.7g
Everything is caged
# echo "::predict" | mdb -k
BP MN KU SZC STATE AGE NEEDED
ffffc1c0089e6000 0 u 4k Disabled 554363 -
ffffc1c008a00000 0 u 2m Disabled 936066 -
ffffc1c008a12000 0 u 1g Inbounds 3598996 -
ffffc1c008400000 0 k 4k Surplus 5 -
ffffc1c0084ee000 0 k 2m Disabled 772189 -
ffffc1c0089d4000 0 k 1g Inbounds 3607468 -
# echo "::breadline" | mdb -k
BREADLINE MN T SZC APD Q'D NEEDED SOUPLINE SATISFIED ABANDONED
ffffe3076a000000 0 u 4k PD - - - 945e4 a285a
ffffe3076a06c000 0 u 2m D - - - 97 605
ffffe3076a0d8000 0 u 1g - - - - -
ffffe3076a000180 0 k 4k - - - da9fa8f 11fbd
ffffe3076a06c180 0 k 2m D - - - 130 1a
ffffe3076a0d8180 0 k 1g - - - - -
Cause
To view full details, sign in with your My Oracle Support account. |
|
Don't have a My Oracle Support account? Click to get started! |
In this Document
| Symptoms |
| Cause |
| Solution |
| References |