My Oracle Support Banner

Kernel panic - not syncing: Hard LOCKUP on distribute_cfs_runtime (Doc ID 2474884.1)

Last updated on FEBRUARY 12, 2019

Applies to:

Linux OS - Version Oracle Linux 7.3 and later
Linux x86-64
Linux x86

Symptoms

 RHCK system HARD LOCKUP panic on Completely Fair Scheduler (CFS) routine distribute_cfs_runtime.

Crash dump analysis shows the following stack trace

crash7lates> sys
KERNEL: /share/linuxrpm/vmlinux_repo/64/3.10.0-514.el7.x86_64/vmlinux
DUMPFILE: All_files/vmcore_hostname [PARTIAL DUMP]
CPUS: 40
DATE: Mon Sep 10 23:47:47 2018
UPTIME: 5 days, 08:06:58
LOAD AVERAGE: 14.17, 10.33, 8.69
TASKS: 1911
NODENAME: hostname
RELEASE: 3.10.0-514.el7.x86_64
VERSION: #1 SMP Thu Nov 3 17:15:05 PDT 2016
MACHINE: x86_64 (2394 Mhz)
MEMORY: 255.6 GB
PANIC: "Kernel panic - not syncing: Hard LOCKUP"
PID: 0
COMMAND: "swapper/3"
TASK: ffff880167423ec0 (1 of 40) [THREAD_INFO: ffff880167434000]
CPU: 3
STATE: TASK_RUNNING (PANIC)

crash7lates> bt
PID: 0 TASK: ffff880167423ec0 CPU: 3 COMMAND: "swapper/3"
#0 [ffff883f7f0c59f0] machine_kexec at ffffffff81059cdb
#1 [ffff883f7f0c5a50] __crash_kexec at ffffffff81105182
#2 [ffff883f7f0c5b20] panic at ffffffff8167f2ba
#3 [ffff883f7f0c5ba0] nmi_panic at ffffffff8108562f
#4 [ffff883f7f0c5bb0] watchdog_overflow_callback at ffffffff8112f0e6
#5 [ffff883f7f0c5bc8] __perf_event_overflow at ffffffff8117465e
#6 [ffff883f7f0c5c00] perf_event_overflow at ffffffff811752a4
#7 [ffff883f7f0c5c10] intel_pmu_handle_irq at ffffffff81009d88
#8 [ffff883f7f0c5e48] perf_event_nmi_handler at ffffffff8168daeb
#9 [ffff883f7f0c5e68] nmi_handle at ffffffff8168ef19
#10 [ffff883f7f0c5eb0] do_nmi at ffffffff8168f093
#11 [ffff883f7f0c5ef0] end_repeat_nmi at ffffffff8168e353
[exception RIP: _raw_spin_lock+58]
RIP: ffffffff8168d81a RSP: ffff883f7f0c3e18 RFLAGS: 00000002
RAX: 0000000000007fe6 RBX: ffff883f799ac9c0 RCX: 000000000000e596
RDX: 000000000000e598 RSI: 000000000000e598 RDI: ffff883f4a720d48
RBP: ffff883f7f0c3e18 R8: 0000000000800008 R9: 0000000000000001
R10: 000000003b9aca00 R11: 0000000000000000 R12: ffff883f77a45800
R13: ffff883f7f956c40 R14: ffff883f4a720c00 R15: ffff883f4a720d48
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
--- <NMI exception stack> ---
#12 [ffff883f7f0c3e18] _raw_spin_lock at ffffffff8168d81a
#13 [ffff883f7f0c3e20] unthrottle_cfs_rq at ffffffff810d100b
#14 [ffff883f7f0c3e58] distribute_cfs_runtime at ffffffff810d12e2
#15 [ffff883f7f0c3ea0] sched_cfs_period_timer at ffffffff810d147f
#16 [ffff883f7f0c3ed8] __hrtimer_run_queues at ffffffff810b4862
#17 [ffff883f7f0c3f30] hrtimer_interrupt at ffffffff810b4e00
#18 [ffff883f7f0c3f80] local_apic_timer_interrupt at ffffffff810510d7
#19 [ffff883f7f0c3f98] smp_apic_timer_interrupt at ffffffff81698bcf
#20 [ffff883f7f0c3fb0] apic_timer_interrupt at ffffffff8169711d
--- <IRQ stack> ---
#21 [ffff880167437e28] apic_timer_interrupt at ffffffff8169711d
[exception RIP: cpu_startup_entry+313]
RIP: ffffffff810e7b89 RSP: ffff880167437ed8 RFLAGS: 00000246
RAX: 0000000000800000 RBX: 7fffffffffffffff RCX: 0100000000000000
RDX: ffff880167437fd8 RSI: 0000000000000000 RDI: 0000000000000002
RBP: ffff880167437f20 R8: 0000000000000000 R9: 0000000000000000
R10: ffff8837d7b4fa50 R11: 0000000000000000 R12: 0000000000000000
R13: ffffffff810f3388 R14: ffff880167437e88 R15: ffffffff810b4782
ORIG_RAX: ffffffffffffff10 CS: 0010 SS: 0018
#22 [ffff880167437f28] start_secondary at ffffffff8104f12a

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


In this Document
Symptoms
Cause
Solution
References


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.