My Oracle Support Banner

Oracle Linux 7 Server Crashed After Experiencing Multiple CPU Soft Lockups with Errors: BUG: soft lockup - CPU#X stuck for XXs! [<process_name>:<pid>] (Doc ID 2622581.1)

Last updated on FEBRUARY 18, 2020

Applies to:

Linux OS - Version Oracle Linux 7.2 with Unbreakable Enterprise Kernel [3.8.13] and later
Linux x86-64

Symptoms

A virtual server running Oracle Linux 7 became non-responsive, and then crashed and was rebooted.

The virtual server was running on VMWare, and the performance statistics for the virtual server showed its CPU utilization was at 100% prior to the crash.

The vmcore buffer log showed the following soft lockup errors occurred immediately prior to the crash and reboot:

[536373.593090] hrtimer: interrupt took 22994736 ns
[546047.980141] [sched_delayed] sched: RT throttling activated
[551100.900567] BUG: soft lockup - CPU#1 stuck for 22s! [top:19389]
[551101.639717] BUG: soft lockup - CPU#2 stuck for 23s! [ora_vktm_gccprd:7742]
[551103.888573] BUG: soft lockup - CPU#3 stuck for 24s! [mpstat:19374]
[551103.888583] BUG: soft lockup - CPU#4 stuck for 25s! [oracle_10670_gc:10670]
[551103.888585] Modules linked in: ksplice_6kox3hlg_vmlinux_new(O)
[551104.205809] Modules linked in:
[551104.205815] ksplice_6kox3hlg(O)

and:

[551128.838231] CPU 7
[551128.838241] Pid: 42, comm: watchdog/7 Tainted: G W O 3.8.13-118.20.2.el7uek.x86_64 #2 VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform
[551128.838244] RIP: 0010:[]
[551131.807021] RIP: 0033:[]
[551132.032144] [] kmem_cache_alloc+0xb1/0x510
[551132.032149] RIP: 0010:[]
[551132.032196] [] 0x7f4663850626
[551132.032201] RIP: 0010:[]
[551132.032210] Pid: 7782, comm: ora_mmnl_gccprd Tainted: G W O 3.8.13-118.20.2.el7uek.x86_64 #2 VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform
[551140.059124] [] ip_rcv_finish+0x9f/0x350
[551140.285667] RSP: 0018:ffff880128f01970 EFLAGS: 00000246
[551140.285680] [] ip_rcv_finish+0x9f/0x350
[551140.285686] RSP: 002b:00007ffef1c5cad8 EFLAGS: 00010202
[551140.285693] ip_tables ext4
[551140.285702] Pid: 10240, comm: oracle_10240_gc Tainted: G W O 3.8.13-118.20.2.el7uek.x86_64 #2 VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform
[551140.285710] Pid: 10275, comm: oracle_10275_gc Tainted: G W O 3.8.13-118.20.2.el7uek.x86_64 #2 VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform
[551140.285715] RSP: 0000:ffff880c3fc83d60 EFLAGS: 00010286
[551140.285722] BUG: soft lockup - CPU#11 stuck for 24s! [jbd2/dm-8-8:1543]
[551140.285792] RAX: 000000000000000f RBX: 00000000ffffffff RCX: 0000000000000010
[551140.285796] RSP: 0000:ffff880c3fce3cf8 EFLAGS: 00010286
[551140.539651] RAX: 00007f4663b8aa50 RBX: 0140000000000001 RCX: 000000007ffffffd
[551140.539657] RIP: 0033:[]
[551180.299017] [] 0x198e6ba
[551180.299023] mbcache

 

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


In this Document
Symptoms
Cause
Solution
References


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.