Lock contention related to ZFS root lock
(Doc ID 1542387.1)
Last updated on JANUARY 16, 2024
Applies to:
Solaris Operating System - Version 10 1/13 U11 to 11.1 [Release 10.0 to 11.0]Oracle Solaris on x86-64 (64-bit)
Oracle Solaris on SPARC (64-bit)
Symptoms
System performance becomes slow. A high percentage of CPU time in kernel ( %sys ) is spent on lock contention.
The ZFS root rwlock will be called once when entering each ZFS mount point. With full path searches
Solaris needs to check every path component to verify if it is a mount point ending up in spending more cpu time in kernel.
vmstat
10:29:28 kthr memory page disk faults cpu
10:29:31 r b w swap free re mf pi po fr de sr s0 s1 s2 s3 in sy cs us sy id
10:29:31 138 0 0 84218616 47104560 238 2544 0 0 0 0 0 1 0 0 0 25208 15315067 3833 26 74 0
10:29:32 160 0 0 84241616 47126656 479 4977 0 0 0 0 0 2 0 3 0 29509 13676089 8380 75 75 0
10:29:34 84 0 0 84245088 47128056 334 3007 0 0 0 0 0 4 0 2 0 26416 16246972 87811 29 71 0
10:29:36 82 0 0 84245672 47128704 463 4589 0 0 0 0 0 1 0 1 0 27704 16219040 332586 28 72 0
10:29:38 72 0 0 84248376 47129312 462 6156 0 0 0 0 0 69 0 67 0 33354 11975297 142132 29 71 0
10:29:40 38 0 0 84217552 47101840 76 5709 0 0 0 0 0 3 0 3 0 37123 2124788 132711 26 74 0
10:29:42 109 0 0 84209800 47093072 123 4058 0 0 0 0 0 9 0 8 0 36321 4320786 71508 24 76 0
10:29:44 117 0 0 84178208 47075712 96 3772 0 0 0 0 0 5 0 4 1 32839 5481426 64172 20 80 0
10:29:46 96 0 0 84172872 47068760 312 4789 0 0 0 0 0 2 0 2 1 32837 3715108 157042 19 81 0
lockstat
Count indv cuml rcnt nsec Lock Caller
311081 15% 15% 0.00 67817 0xffffff115b67de50 rrw_exit+0x1d
nsec ----- Time Distribution ----- count Stack
256 | 922 zfs_root+0x78
512 |@ 14133 fsop_root+0x2d
1024 |@@@ 37051 traverse+0x65
2048 |@@@@@@@ 81174 lookuppnvp+0x446
4096 |@@@@@@ 4803
8192 |@@@ 36934
16384 |@ 17009
32768 |@ 11281
65536 | 5997
131072 | 7786
262144 | 227
524288 |@ 11320
1048576 | 9317
2097152 | 3712
4194304 | 413
8388608 |
Count indv cuml rcnt nsec Lock Caller
303613 15% 29% 0.00 90197 0xffffff115b67de50 rrw_enter_read+0x1b
nsec ----- Time Distribution ----- count Stack
256 | 483 rrw_enter+0x2c
512 |@ 10995 zfs_root+0x3b
1024 |@ 20181 fsop_root+0x2d
2048 |@@@@@ 54923 traverse+0x65
4096 |@@@@@@@ 79387
8192 |@@@@ 4601
16384 |@ 20205
32768 |@ 13852
65536 | 7695
131072 | 9993
262144 |@ 11829
524288 |@ 14885
1048576 |@ 12059
2097152 | 5054
4194304 | 469
Count indv cuml rcnt nsec Lock Caller
632033 100% 100% 0.00 72575 0xffffff115b67df60 zfs_zget+0x46
nsec ------ Time Distribution ------ count Stack
256 | 936 zfs_root+0x55
512 | 15849 fsop_root+0x2d
1024 |@ 36944 traverse+0x65
2048 |@@@ 72479 lookuppnvp+0x446
Changes
Upgrading to Solaris 10 U11 (OR) later
Cause
To view full details, sign in with your My Oracle Support account. |
|
Don't have a My Oracle Support account? Click to get started! |
In this Document
Symptoms |
Changes |
Cause |
Solution |
References |