panic in dnode_dest() due to mutex_destroy: not owner (Doc ID 2413454.1)

Last updated on JULY 06, 2023

Applies to:

Symptoms

The symptom you will see is a panic in mutex_destroy() where your thread is not the owner of the mutex:

     mutex_destroy: not owner, lp=fff8011864367a68 owner=c40112e09580 thread=2a100009b80

and your panic stack will appear similar to the following ( from system messages) :

...
...
Apr 30 05:20:42 hostname ^Mpanic[cpu26]/thread=2a100009b80:
Apr 30 05:20:42 hostname unix: [ID 103648 kern.notice] mutex_destroy: not owner, lp=fff8011864367a68 owner=c40112e09580 thread=2a100009b80
Apr 30 05:20:42 hostname unix: [ID 100000 kern.notice]
Apr 30 05:20:42 hostname genunix: [ID 723222 kern.notice] 000002a1000095e0 unix:mutex_destroy+120 (10122000, fff8011864367a68, 28, 24, 0, 208a9690)
Apr 30 05:20:42 hostname genunix: [ID 179002 kern.notice] %l0-3: 0000000000000000 0000000000000001 0000c40112e09580 0000000000000080
Apr 30 05:20:42 hostname %l4-7: 00000000208a9400 0000c40112e09580 0000c40112e09580 0000000000000000
Apr 30 05:20:42 hostname genunix: [ID 723222 kern.notice] 000002a100009690 zfs:dnode_dest+c (fff8011864367990, 0, fff8011864367996, 4, 2a100009b84, 2a100009b64)
Apr 30 05:20:42 hostname genunix: [ID 179002 kern.notice] %l0-3: 0000000000000000 0000000000000000 0000000000000000 fff801186436799c
Apr 30 05:20:42 hostname %l4-7: 000000000000000c fff8011864367a70 0000000000000004 0000000000000000
Apr 30 05:20:42 hostname genunix: [ID 723222 kern.notice] 000002a100009740 genunix:kmem_slab_free_constructed+c4 (300019c0000, fff8011864367990, 0, 1, fff8011864367bd0, 0)
Apr 30 05:20:42 hostname genunix: [ID 179002 kern.notice] %l0-3: 00000000208fb800 0000c40111ff5540 000000001205e350 0000000000000001
Apr 30 05:20:42 hostname %l4-7: 0000000000000060 0000000000000001 7fffffffffffffff 8000000000000000
Apr 30 05:20:42 hostname genunix: [ID 723222 kern.notice] 000002a1000097f0 genunix:kmem_move_buffer+1bc (fff8011864367990, 3000c457a78, 300019c0000, 300019d3ed0, 0, fff8011864367fb8)
Apr 30 05:20:42 hostname genunix: [ID 179002 kern.notice] %l0-3: 00000300019c0120 00000300019d3ed0 00000300019d3ed0 00000300019d3ed0
Apr 30 05:20:42 hostname %l4-7: 000000001205f098 00000300019d3ed0 00000300019d3ed0 fff801022f1f9d00
Apr 30 05:20:42 hostname genunix: [ID 723222 kern.notice] 000002a1000098a0 genunix:taskq_thread+42c (2ab2858623c7c1, c40083c65ea0, 2ab2858623ca7b, c40083c65ed2, c40083c65ed4, c400a6dbe5e8)
Apr 30 05:20:42 hostname genunix: [ID 179002 kern.notice] %l0-3: 0000000000080000 0000000000010000 0000c40083c65ed0 0000000000000001
Apr 30 05:20:42 hostname %l4-7: 0000c40083c65ec0 0000c40083c65f10 0000c40083c65ec8 00000000fffeffff
...
...

Kmem is in the process of moving a buffer via kmem_move_buffer() and has finished with a device node (dnode) and wants to destroy it.  A review of the crash dump should show the arc_evictor_thread is releasing the dnode and has taken over the ownership of the mutex. Its taking its time to release the mutex and the panic triggers.

Cause

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.