My Oracle Support Banner

Kernel Panic at [Exception RIP: blk_mq_check_expired+24] (Doc ID 2587893.1)

Last updated on OCTOBER 17, 2019

Applies to:

Linux OS - Version Oracle Linux 6.7 with Unbreakable Enterprise Kernel [4.1.12] and later
Oracle Cloud Infrastructure - Version N/A and later
Linux x86-64

Symptoms

Systems can suffer random NULL-pointer panics when blk-mq handles timed out requests.

A typical panic string can look like this:

PANIC: "BUG: unable to handle kernel NULL pointer dereference at 0000000000000450"

A typical panic stack can look like this:

crash7latest> bt
PID: 2327 TASK: ffff880fc0438e00 CPU: 7 COMMAND: "kworker/7:1H"
#0 [ffff880fc0e47960] machine_kexec at ffffffff81062a6b
#1 [ffff880fc0e479d0] crash_kexec at ffffffff8111cff2
#2 [ffff880fc0e47aa0] oops_end at ffffffff8101b938
#3 [ffff880fc0e47ad0] no_context at ffffffff81747c5c
#4 [ffff880fc0e47b30] __bad_area_nosemaphore at ffffffff81747d44
#5 [ffff880fc0e47b80] bad_area_nosemaphore at ffffffff81747eb0
#6 [ffff880fc0e47b90] __do_page_fault at ffffffff810723a6
#7 [ffff880fc0e47c00] trace_do_page_fault at ffffffff81072869
#8 [ffff880fc0e47c40] do_async_page_fault at ffffffff8106b3de
#9 [ffff880fc0e47c60] async_page_fault at ffffffff8175be4f
[exception RIP: blk_mq_check_expired+24]
RIP: ffffffff81327aa8 RSP: ffff880fc0e47d18 RFLAGS: 00010246
RAX: 0000000000000000 RBX: 0000000000000014 RCX: 0000000000000000
RDX: ffff880fc0e47dc0 RSI: ffff880fc25ab800 RDI: ffff880fc25ab800
RBP: ffff880fc0e47d18 R8: ffff880fc0e47dc0 R9: 0000000000000001
R10: 0000000000001111 R11: 0000000000000000 R12: 0000000000000005
R13: ffff880fc1270b50 R14: ffff880fc1396000 R15: ffff880fc2f86d40
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
#10 [ffff880fc0e47d20] bt_for_each at ffffffff8132a177
#11 [ffff880fc0e47d80] blk_mq_tag_busy_iter at ffffffff8132a1d6
#12 [ffff880fc0e47db0] blk_mq_timeout_work at ffffffff813262c3
#13 [ffff880fc0e47e00] process_one_work at ffffffff810a7025
#14 [ffff880fc0e47e50] worker_thread at ffffffff810a7832
#15 [ffff880fc0e47ec0] kthread at ffffffff810ad4aa
#16 [ffff880fc0e47f50] ret_from_fork at ffffffff81756a41
crash7latest>

A typical pattern in dmesg can look like:

[695101.496601] BUG: unable to handle kernel NULL pointer dereference at 0000000000000450
[695101.498435] IP: [<ffffffff81327aa8>] blk_mq_check_expired+0x18/0x90
[695101.499861] PGD 8000000809e83067 PUD 8171e2067 PMD 0
[695101.501101] Oops: 0000 [#1] SMP
[695101.501911] Modules linked in: sunrpc ppdev crct10dif_pclmul crc32_pclmul ghash_clmulni_intel dm_mod aesni_intel lrw gf128mul glue_helper ablk_helper cryptd parport_pc parport i2c_piix4 acpi_cpufreq i2c_core pcspkr binfmt_misc ip_tables xfs libcrc32c crc32c_intel serio_raw nvme ena nvme_core
[695101.508491] CPU: 7 PID: 2327 Comm: kworker/7:1H Not tainted 4.1.12-124.25.1.el7uek.x86_64 #2
[695101.510343] Hardware name: Amazon EC2 r5.2xlarge/, BIOS 1.0 10/16/2017
[695101.511796] Workqueue: kblockd blk_mq_timeout_work
[695101.512915] task: ffff880fc0438e00 ti: ffff880fc0e44000 task.ti: ffff880fc0e44000
[695101.514559] RIP: 0010:[<ffffffff81327aa8>] [<ffffffff81327aa8>] blk_mq_check_expired+0x18/0x90

Changes

N/A

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


In this Document
Symptoms
Changes
Cause
Solution
References


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.