mutex_enter: bad mutex panic at rpcib:rib_clnt_scq_handler (Doc ID 2239943.1)

Last updated on MARCH 03, 2017

Applies to:

Solaris Operating System - Version 11.3 and later
Information in this document applies to any platform.

Symptoms

A sudden kernel panic will cause the node to reboot without warning.

Solaris OS version is below 11.3.6.5.0.

The /var/adm/messages file will show a panic entry similar to:

Feb 15 14:56:45 cbrtpstdb101 ^Mpanic[cpu41]/thread=2a105829b80:
Feb 15 14:59:01 cbrtpstdb101 ssctuner: [ID 702911 local0.notice] notice: enabled, version 1.4.1. daemon PID= 752
Feb 15 14:56:45 cbrtpstdb101 unix: [ID 103648 kern.notice] mutex_enter: bad mutex, lp=30403553bde40 owner=3040365172200 thread=2a105829b80
Feb 15 14:59:03 cbrtpstdb101 ssctuner: [ID 702911 local0.error] critical: Please change ssctuner email address from root@localhost
Feb 15 14:56:45 cbrtpstdb101 unix: [ID 100000 kern.notice]
Feb 15 14:59:03 cbrtpstdb101 ssctuner: [ID 702911 local0.warning] warning: Manual NFS tuning needed. See /opt/oracle.supercluster/ssctuner/nfs_advice.txt
Feb 15 14:56:45 cbrtpstdb101 genunix: [ID 723222 kern.notice] 000002a105829240 rpcib:rib_clnt_scq_handler+80 (304018bee7508, 4d3a00000000, 1d4d3a1d, 30402529c9c00, 2a1058292f8, 1)
Feb 15 14:59:04 cbrtpstdb101 ssctuner: [ID 702911 local0.error] critical: SSCTUNER pkg is over 12 months old - UPDATE NOW
Feb 15 14:56:45 cbrtpstdb101 genunix: [ID 179002 kern.notice] %l0-3: 0000000000000014 00030403553bde40 00030403553bde38 0000000000000008
Feb 15 14:59:04 cbrtpstdb101 ssctuner: [ID 702911 local0.notice] notice: Checking Oracle log writer and LMS thread priority.
Feb 15 14:56:45 cbrtpstdb101 %l4-7: 00004d3a1d4d3a1d 0000000000000000 0000000000000000 000002a1073cf110
Feb 15 14:59:04 cbrtpstdb101 ssctuner: [ID 702911 local0.notice] notice: completed initialization.
Feb 15 14:56:45 cbrtpstdb101 genunix: [ID 723222 kern.notice] 000002a105829340 ibtl:ibc_cq_handler+8c (304018bee7528, 304018bee7508, 0, 110065a8, c8ffffff, 20922800)
Feb 15 14:56:45 cbrtpstdb101 genunix: [ID 179002 kern.notice] %l0-3: 0000000000000000 00000000000186a0 0000000030000000 0000000010000000
Feb 15 14:56:45 cbrtpstdb101 %l4-7: 0000000002db12c5 0000000020923800 0000000010000000 0000000010000000
Feb 15 14:56:45 cbrtpstdb101 genunix: [ID 723222 kern.notice] 000002a1058293f0 mcxnex:mcxnex_cq_handler+60 (304015fd38000, 3ffff, 304014f087c80, aad, ffffff, fffc00)
Feb 15 14:56:45 cbrtpstdb101 genunix: [ID 179002 kern.notice] %l0-3: 00000000208a240c 00000000204cc800 0000000020121c18 000000000000000b
Feb 15 14:56:45 cbrtpstdb101 %l4-7: 000304014f0ea6c8 0000000000000aad 0000000000000aad 0000000000000aad
Feb 15 14:56:45 cbrtpstdb101 genunix: [ID 723222 kern.notice] 000002a1058294b0 mcxnex:mcxnex_eq_poll+d8 (304015fd38000, 304014f039e38, 80, 1, ffffff, 6)
Feb 15 14:56:45 cbrtpstdb101 genunix: [ID 179002 kern.notice] %l0-3: 00000000970febef 0000000000003978 000304015fd38b70 0000000000000001
Feb 15 14:56:45 cbrtpstdb101 %l4-7: 000002a1058295ac 0000040029b7fde0 0000000000000001 0000018200c65a29
Feb 15 14:56:45 cbrtpstdb101 genunix: [ID 723222 kern.notice] 000002a1058295b0 mcxnex:mcxnex_isr+188 (304015fd38000, 1d, 1d, 0, 1c, 4)
Feb 15 14:56:45 cbrtpstdb101 genunix: [ID 179002 kern.notice] %l0-3: 0000000000000000 00000000000003fe 000004001749f7d8 0000000000000000
Feb 15 14:56:45 cbrtpstdb101 %l4-7: 000304015fd380e8 0000000000001170 0000000000001000 00000000000000e8
Feb 15 14:56:45 cbrtpstdb101 genunix: [ID 723222 kern.notice] 000002a105829670 pxsoft:pxsoft_run_evnt_hdl+118 (30401a8f21780, 2a105829840, 30401a8f21780, 0, 1, 2a105829840)
Feb 15 14:56:45 cbrtpstdb101 genunix: [ID 179002 kern.notice] %l0-3: 0000000000000000 0000000000000000 0000000020121c18 000002a10736fb80
Feb 15 14:56:45 cbrtpstdb101 %l4-7: 0000000000000003 0000000000000002 0000000000000000 0000000000000100
Feb 15 14:56:45 cbrtpstdb101 genunix: [ID 723222 kern.notice] 000002a105829730 priq:priq_event_handler+1b8 (304015e3f2340, 11814474, 1, 6, 100000100000031, 30401a8f21780)
Feb 15 14:56:45 cbrtpstdb101 genunix: [ID 179002 kern.notice] %l0-3: 0003040192bdb2c0 0000000000000000 fffffffffffffffe 0000000000000001
Feb 15 14:56:45 cbrtpstdb101 %l4-7: 00030401a8f21780 0000000000000003 0000000000000301 00000000000003fe
Feb 15 14:56:45 cbrtpstdb101 genunix: [ID 723222 kern.notice] 000002a105829880 unix:dispatch_handler+220 (30000054000, 2a105829b80, 0, 700900c0, 6, 1)
Feb 15 14:56:45 cbrtpstdb101 genunix: [ID 179002 kern.notice] %l0-3: 0000030000054060 0000000000000040 0000030000054028 0000030000054030
Feb 15 14:56:45 cbrtpstdb101 %l4-7: 0000000000000000 0000000011818cf8 000000000000000f 000304015e3f2340

 

Analysis of vmcore will show a panic stack matching the one in bug 22082022.

 

 

For other bad mutex related panics, see Doc ID 2239005.1 and Doc ID 2197583.1.

 

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms