My Oracle Support Banner

Performance problems on LDOM due to locking contention in cnex driver (Doc ID 2204804.1)

Last updated on JUNE 22, 2017

Applies to:

Solaris Operating System - Version 10 11/06 U3 and later
Information in this document applies to any platform.

Symptoms

When vmstat(1) and mpstat(1) show a high CPU sys percentage and mpstat(1) shows a high number of spins on mutexes (column smtx)...

CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  st idl
  0   40   0 2572  4056 1012 5712    8  306  477    0  2127    2   8   0  90
  1  145   0 2029  3207  617 5181    8  325  199    0   736    1   7   0  92
  2  126   0 3528  4962 2320 5388    9  327  335    0   158    0   8   0  92
  3  297   0 2191  3601  903 5432   10  306  127    0   534    1   7   0  91
  4   73   0 2530  4276 1520 5506    6  309  195    0   166    0   7   0  93
  5  283   0 2748  3930 1202 5582    9  289  125    0   346    0   7   0  92
  6   67   0 3161  4366 1797 5305    7  310  168    0   232    0   8   0  92
  7   90   0 2368  3230  594 5236    9  281  504    0  2972    2   9   0  90
CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  st idl
  0 1185   0 5380  6892  978 11275  480 1506 3308    0  3226   19  28   0  54
  1 1291   0 4210  5971 1021  9762  504 1477 1024    0  2240   25  25   0  50
  2 1794   0 6196  7976 2194 11890  677 1839 1029    0  2407   16  30   0  53
  3 1601   0 6397  8234 2248 13133  812 1888  972    0  2681   22  24   0  53
  4 1537   0 6155  7422 1879 12036  700 1730 1890    0  2364   24  24   0  52
  5 1489   0 5920  6990 1492 11646  700 1647 1840    0  2677   23  26   0  51
  6 1762   0 5139  6452  832 11389  618 1644  902    0  3178   25  24   0  50
  7 1427   0 4903  6118  623 11648  733 1733 1792    0  5973   26  23   0  50

...and lockstat(1) shows a lock contention in function cnex_clr_intr()...

Adaptive mutex spin: 5812 events in 2.067 seconds (2811 events/sec)

-------------------------------------------------------------------------------
Count indv cuml rcnt     nsec Lock                   Hottest Caller
  294  49%  49% 0.00  1097287 0x40003dd4518          cnex_clr_intr+0x28

      nsec ------ Time Distribution ------ count     Stack
       128 |                               8         ldc_read+0x174
       256 |                               7         recv_msg+0x2c
       512 |@@                             26        vd_recv_msg+0x54
      1024 |@@@                            39        taskq_thread+0x3e0
      2048 |@@@@@@                         66        thread_start+0x4
      4096 |@@@                            35
      8192 |@                              16
     16384 |@@                             22
     32768 |                               5
     65536 |                               5
    131072 |                               1
    262144 |                               0
    524288 |                               0
   1048576 |                               0
   2097152 |@                              13
   4194304 |@                              18
   8388608 |@@                             23
  16777216 |@                              10
-------------------------------------------------------------------------------
Adaptive mutex block: 319 events in 2.067 seconds (154 events/sec)

-------------------------------------------------------------------------------
Count indv cuml rcnt     nsec Lock                   Hottest Caller
   25  55%  55% 0.00  2415310 0x40003dd4518          cnex_clr_intr+0x28

      nsec ------ Time Distribution ------ count     Stack
      8192 |@@@                            3         ldc_read+0x174
     16384 |@                              1         recv_msg+0x2c
     32768 |@@@                            3         vd_recv_msg+0x54
     65536 |@@@@@@                         5         taskq_thread+0x3e0
    131072 |@                              1         thread_start+0x4
    262144 |                               0
    524288 |                               0
   1048576 |                               0
   2097152 |@@@@                           4
   4194304 |@@                             2
   8388608 |@@@@                           4
  16777216 |@@                             2
-------------------------------------------------------------------------------
Adaptive mutex spin: 10624 events in 2.066 seconds (5141 events/sec)

-------------------------------------------------------------------------------
Count indv cuml rcnt     nsec Lock                   Hottest Caller
  173  17%  17% 0.00   656308 0x40003dd4518          cnex_clr_intr+0x28

      nsec ------ Time Distribution ------ count     Stack
       128 |                               1         ldc_read+0x174
       256 |@                              7         recv_msg+0x2c
       512 |@@                             14        vd_recv_msg+0x54
      1024 |@@@@                           28        taskq_thread+0x3e0
      2048 |@@@@@@                         36        thread_start+0x4
      4096 |@@@@                           28
      8192 |@@                             16
     16384 |@@                             12
     32768 |@                              6
     65536 |                               0
    131072 |                               0
    262144 |                               0
    524288 |                               0
   1048576 |                               2
   2097152 |                               5
   4194304 |@                              11
   8388608 |                               3
  16777216 |                               4
-------------------------------------------------------------------------------

...then the system is hitting a known and fixed issue in the cnex driver.

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


In this Document
Symptoms
Cause
Solution
References


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.