Lock contention related to ZFS root lock (Doc ID 1542387.1)

Last updated on JANUARY 16, 2024

Applies to:

Symptoms

System performance becomes slow. A high percentage of CPU time in kernel ( %sys )  is spent on lock contention.

The ZFS root rwlock will be called once when entering each ZFS mount point. With full path searches

Solaris needs to check every path component to verify if it is a mount point ending up in spending more cpu time in kernel.

vmstat

10:29:28   kthr      memory            page                 disk              faults                                     cpu

10:29:31   r b w   swap  free  re  mf pi po fr de sr s0 s1 s2 s3   in   sy   cs                             us   sy id
10:29:31   138 0 0 84218616 47104560 238 2544 0 0 0 0 0 1 0 0  0 25208 15315067 3833      26  74  0
10:29:32   160 0 0 84241616 47126656 479 4977 0 0 0 0 0 2 0 3  0 29509 13676089 8380      75  75  0
10:29:34   84 0 0 84245088 47128056 334 3007 0 0 0 0 0 4 0  2  0 26416 16246972 87811     29  71  0
10:29:36   82 0 0 84245672 47128704 463 4589 0 0 0 0 0 1 0  1  0 27704 16219040 332586   28  72  0
10:29:38   72 0 0 84248376 47129312 462 6156 0 0 0 0 0 69 0 67 0 33354 11975297 142132  29  71  0
10:29:40   38 0 0 84217552 47101840 76 5709 0 0 0 0 0 3  0  3  0 37123 2124788 132711      26  74  0
10:29:42   109 0 0 84209800 47093072 123 4058 0 0 0 0 0 9 0 8  0 36321 4320786 71508      24  76  0
10:29:44   117 0 0 84178208 47075712 96 3772 0 0 0 0 0 5 0  4  1 32839 5481426 64172       20  80  0
10:29:46   96 0 0 84172872 47068760 312 4789 0 0 0 0 0 2 0  2  1 32837 3715108 157042     19  81  0

lockstat
 
 Count  indv cuml rcnt    nsec      Lock                  Caller

 311081  15%  15% 0.00    67817 0xffffff115b67de50     rrw_exit+0x1d

       nsec ----- Time Distribution ----- count                Stack
        256 |                                       922     zfs_root+0x78
        512 |@                                    14133     fsop_root+0x2d
       1024 |@@@                                  37051     traverse+0x65
       2048 |@@@@@@@                              81174     lookuppnvp+0x446
       4096 |@@@@@@                                4803
      8192  |@@@                                  36934
      16384 |@                                    17009
      32768 |@                                    11281
      65536 |                                      5997
     131072 |                                      7786
     262144 |                                       227
     524288 |@                                    11320
    1048576 |                                      9317
    2097152 |                                      3712
    4194304 |                                       413
    8388608 |                                       


 Count  indv cuml rcnt    nsec       Lock                   Caller

 303613  15%  29% 0.00    90197 0xffffff115b67de50     rrw_enter_read+0x1b  

       nsec ----- Time Distribution ----- count               Stack
        256 |                                      483     rrw_enter+0x2c
        512 |@                                   10995     zfs_root+0x3b
       1024 |@                                   20181     fsop_root+0x2d
       2048 |@@@@@                               54923     traverse+0x65
       4096 |@@@@@@@                             79387
       8192 |@@@@                                 4601
      16384 |@                                   20205
      32768 |@                                   13852
      65536 |                                     7695
     131072 |                                     9993
     262144 |@                                   11829
     524288 |@                                   14885
    1048576 |@                                   12059
    2097152 |                                     5054
    4194304 |                                      469

 
 Count indv cuml rcnt    nsec          Lock                   Caller

 632033 100% 100% 0.00    72575 0xffffff115b67df60     zfs_zget+0x46 

       nsec ------ Time Distribution ------ count           Stack
        256 |                                          936       zfs_root+0x55
        512 |                                        15849     fsop_root+0x2d
       1024 |@                                      36944     traverse+0x65
       2048 |@@@                                 72479     lookuppnvp+0x446

Changes

Upgrading to Solaris 10 U11 (OR) later

Cause

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.