Solaris Operating System Hang or Panic due to Third Party Kernel Modules (Doc ID 1411793.1)

Last updated on NOVEMBER 02, 2017

Applies to:

Symptoms

Solaris System may hang or panic in a Third Party Module or the panic may be related to a Third Party Module.

Third Party in this instance refers to a kernel Module that is not provided with the Solaris Operating Environment.

User level applications operate out side of the kernel and should not be able to cause the kernel to panic the system.  Kernel modules are run within the kernel and are given full access and permission to the kernel memory space which means they are fully capable of causing the kernel to panic.

For an example, the following panic was recorded in the kernel's message buffer.
This should also closely match what would be observed in a system's messages file:

The following was attained by using:
echo "::msgbuf" | mdb -k unix.0 vmcore.0
...
...
panic[cpu173]/thread=2a1038c1ca0:
BAD TRAP: type=31 rp=2a1038c0cc0 addr=10 mmu_fsr=0 occurred in module "vxfs" due to a NULL pointer dereference

sched:
trap type = 0x31
addr=0x10
pid=0, pc=0x7aa39644, sp=0x2a1038c0561, tstate=0x80001603, context=0x0
g1-g7: ad, 152, 3c00, 800, 1, 16, 2a1038c1ca0

000002a1038c09e0 unix:die+78 (31, 2a1038c0cc0, 10, 0, 2a1038c0aa0, 10a4c00)
%l0-3: 0000000000001fff 0000000000000031 0000000001000000 0000000000002000
%l4-7: 000000000183a368 000000000183a000 0000000000000001 00000000ea5da010
000002a1038c0ac0 unix:trap+9e0 (2a1038c0cc0, 0, 1fff, 5, 0, 1)
%l0-3: 0000000000000000 00000000018928c0 0000000000000031 0000000000001c00
%l4-7: 0000000000000000 0000000000000001 ffffffffffffe000 0000000000000005
000002a1038c0c10 unix:ktl0+48 (0, 930, 2000, ee7c2a5, ee7be63, 600f0b27c00)
%l0-3: 0000000000000004 0000000000001400 0000000080001603 000000000101c9fc
%l4-7: 0000000000000004 00000000000000ad 0000000000000000 000002a1038c0cc0
000002a1038c0d60 600f2ca2038 (600fe86a900, 600f2ca2038, 7c2, 3e, 600ff644000, 0)
%l0-3: 0000000000000000 000000000004bb00 00000600ca900000 0000000000000000
%l4-7: 000000000ee7c2a5 0000000000000568 0000000000000000 00000600fe86a920
000002a1038c0e10 vxfs:vx_tflush_map+138 (600ff644000, 600fe86a900, 2000, 2, ad, 3000f904000)
%l0-3: 00000000702f1180 00000600ff644718 0000000000000002 0000000000000000
%l4-7: 0000000000000000 0000000000000000 00000000000004bb 0000000000000930
000002a1038c0ef0 vxfs:vx_fsq_flush+4d8 (0, 0, 0, 600ff644000, 0, 702f1180)
%l0-3: 0000000031e3bc66 0000000000000002 00000600ff644510 0000000000000000
%l4-7: 00000600ff644438 0000000000000000 0000000000000002 0000000000000000
000002a1038c1020 vxfs:vx_tranflush+14 (600ff644000, 1, 2, 0, 67bdb, 67bda)
%l0-3: 0000000000000000 00000600ca94bb00 00000600ca900000 000000000004bb00
%l4-7: 00000000000004bb 00000000000000ad 0000000000000000 0000000000000000
000002a1038c10d0 vxfs:vx_traninit+320 (600ff644000, 1ff, 21, 3, 0, 4bb)
%l0-3: 0000000001833500 0000000051eb8400 0000000000000000 0000000000000001
%l4-7: 00000000702f1180 000003000f904000 00000000000000ad 00000600ca900000
000002a1038c1190 vxfs:vx_trunc_tran2+4dc (600faea7b80, 30096e42550, 0, 0, 1, 0)
%l0-3: 0000000000000001 00000600ff644000 0000000000000000 0000030096e42938
%l4-7: 0000000000000000 0000000000000001 0000000000000004 0000000000000000
000002a1038c1370 vxfs:vx_trunc_tran+158 (600faea7b80, 30096e42550, 0, 2a575, 1, 0)
%l0-3: 000002a1038c1440 0000000000000000 0000000000000000 0000000000000000
%l4-7: 000000000002a575 0000000000000004 0000030096e42930 0000000000000000
000002a1038c1480 vxfs:vx_trunc+27c (0, 81a4, 0, f000, 6485c, 0)
%l0-3: 00000600ff6441f8 0000000000000003 000000000002a575 000000007ab23730
%l4-7: 00000000703c07f0 00000600f54a50a0 0000000000000000 0000030096e42550
000002a1038c1560 vxfs:vx_inactive_remove+e8 (30096e42550, 8000, 81a4, 0, ff00f000, 0)
%l0-3: 0000000000000000 0000000000000000 0000000000000000 00000600ff644000
%l4-7: 00000600faea7b80 0000000000000000 0000000000000001 0000000000000000
000002a1038c1630 vxfs:vx_inactive_tran+5e4 (30096e42550, 0, 4, 1, 600f54a50a0, 0)
%l0-3: 00000600ff644000 0000000000000001 0000000000000000 0000000000000001
%l4-7: 0000000000000000 00000000000081a4 0000000000000000 000000001400a400
000002a1038c1710 vxfs:vx_local_inactive_list+8 (30096e42550, 2a1038c1ca0, 600ca900000, 702f1000, 702f1000, 7ab063ac)
%l0-3: 000000000000001c 0000000000005680 0000000000000000 0000000000000000
%l4-7: 000006019429f6b0 000006019429f680 0000000000000039 0000000000000038
000002a1038c17c0 vxfs:vx_inactive_list+470 (601459a30a8, 600f54a50a0, 1, 0, 7, 10)
%l0-3: 00000600caa1c5d8 00000600caa9c580 0000000000000001 0000000000000000
%l4-7: 00000000702f1230 00000000702f1698 0000000010000000 0000030096e42550
000002a1038c1880 vxfs:vx_workitem_process+10 (60179431c98, 28, 0, 5, 1, 7ab05e70)
%l0-3: 00000000019437b8 000003000f9042e0 0000000000000000 0000000000000000
%l4-7: 000002a102ce1ca0 0000000000000003 0000000001833048 0000000001833040
000002a1038c1930 vxfs:vx_worklist_process+17c (1, 0, 0, 702efb40, 601755715a0, 702efe50)
%l0-3: 00000000702efb18 0000000000000001 00000000702efb40 00000000702efe54
%l4-7: 00000000702efb40 00000000702f0ee0 00000000702efb40 0000060179431c98
000002a1038c19e0 vxfs:vx_worklist_thread+94 (22, 702f1400, 702f1400, 702f1634, 702f1718, 0)
%l0-3: 0000000000000000 0000000000000001 0000000000000000 00000000702f0880
%l4-7: 0000000000000082 0000000000000000 00000000702f03d0 0000000000000000
...

The above panic happened in the module "vxfs" and all of the other functions are also from the "vxfs" module except for the top three which are from the module "unix". Note that the module unix is part of the main kernel, and those function are error handling code which handled the error and triggered the panic to provide the kernel crash dump to allow an analysis. 

Look though the functions involved shows that each function contains a module followed by a colon. This was followed by a function and  the offset within that function where it left that function for the next one. After this are some possible arguments enclosed in parenthesis to be held in the cpu registers.  Note that a stack is read from the bottom up. Additional discussion about the stack is beyond the scope of the article.

The "vxfs" module may be familiar.  If not, more information can be obtained from crash dump .

$ echo "::modctl" | mdb -k unix.0 vmcore.0 | grep vxfs
600c48c0230 600cad19cc0 li 0x00 2 fs/vxfs
$ echo "::modinfo" | mdb -k unix.0 vmcore.0 | grep vxfs
228 7aa00000 1bc8a0 1 vxfs (VxFS 5.0_REV-5.0MP1u_sol SunOS )

This shows us that the vxfs module is part of a the VxFS 5.0 package.  If this is checked on the Internet this package is supported by the Symantec Corporation in this case.  Other modules may be supported by different Third Party Vendors.

Cause