My Oracle Support Banner

Oracle Linux: OL7 VM/DomU Reboots Abruptly while Running DB Indexing Job with Error "BUG:soft lockup - CPU#0 stuck for 22s" (Doc ID 2614748.1)

Last updated on FEBRUARY 26, 2020

Applies to:

Linux OS - Version Oracle Linux 7.5 with Unbreakable Enterprise Kernel [4.14.35] and later
Linux x86-64

Symptoms

VM reboots when DB indexing job is started. The following call trace is observed in /var/log/messages:

Nov 18 19:38:24 <HOSTNAME> kernel: [ 1252.092275] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [swapper/0:0]
Nov 18 19:38:24 <HOSTNAME> kernel: [ 1252.092315] Modules linked in: nfsv3 nfs_acl nfs lockd grace fscache ovmapi sunrpc xfs libcrc32c ext4 mbcache jbd2 fscrypto dm_mod ppdev xen_fbfront crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc cirrus aesni_intel ttm crypto_simd glue_helper cryptd drm_kms_helper parport_pc pcspkr parport drm syscopyarea sysfillrect sysimgblt fb_sys_fops i2c_piix4 binfmt_misc ip_tables btrfs xor zstd_decompress zstd_compress xxhash raid6_pq xen_netfront ata_generic pata_acpi xen_blkfront ata_piix libata crc32c_intel serio_raw floppy
Nov 18 19:38:24 <HOSTNAME> kernel: [ 1252.092359] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G L 4.14.35-1902.6.6.el7uek.x86_64 #2
Nov 18 19:38:24 <HOSTNAME> kernel: [ 1252.092360] Hardware name: Xen HVM domU, BIOS 4.4.4OVM 11/16/2018
Nov 18 19:38:24 <HOSTNAME> kernel: [ 1252.092362] task: ffffffff90412480 task.stack: ffffffff90400000
Nov 18 19:38:24 <HOSTNAME> kernel: [ 1252.092371] RIP: 0010:native_safe_halt+0x12/0x14
Nov 18 19:38:24 <HOSTNAME> kernel: [ 1252.092372] RSP: 0018:ffffffff90403e18 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff0c
Nov 18 19:38:24 <HOSTNAME> kernel: [ 1252.092374] RAX: ffffffff8f86be20 RBX: ffffffff90412480 RCX: 0000000000000000
Nov 18 19:38:24 <HOSTNAME> kernel: [ 1252.092375] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Nov 18 19:38:24 <HOSTNAME> kernel: [ 1252.092376] RBP: ffffffff90403e18 R08: ffffade118597a28 R09: ffffade114477e88
Nov 18 19:38:24 <HOSTNAME> kernel: [ 1252.092377] R10: ffffade10f093cb0 R11: 000000000000001a R12: 0000000000000000
Nov 18 19:38:24 <HOSTNAME> kernel: [ 1252.092378] R13: ffffffff90412480 R14: 0000000000000000 R15: 0000000000000000
Nov 18 19:38:24 <HOSTNAME> kernel: [ 1252.092380] FS: 0000000000000000(0000) GS:ffff98a48d600000(0000) knlGS:0000000000000000
Nov 18 19:38:24 <HOSTNAME> kernel: [ 1252.092381] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 18 19:38:24 <HOSTNAME> kernel: [ 1252.092383] CR2: 00000000afc604f8 CR3: 000000038c40a006 CR4: 00000000001606f0
Nov 18 19:38:24 <HOSTNAME> kernel: [ 1252.092386] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Nov 18 19:38:24 <HOSTNAME> kernel: [ 1252.092387] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Nov 18 19:38:24 <HOSTNAME> kernel: [ 1252.092387] Call Trace:
Nov 18 19:38:24 <HOSTNAME> kernel: [ 1252.092392] default_idle+0x1e/0xfc
Nov 18 19:38:24 <HOSTNAME> kernel: [ 1252.092397] arch_cpu_idle+0x15/0x17
Nov 18 19:38:24 <HOSTNAME> kernel: [ 1252.092398] default_idle_call+0x23/0x31
Nov 18 19:38:24 <HOSTNAME> kernel: [ 1252.092404] do_idle+0x158/0x1d5
Nov 18 19:38:24 <HOSTNAME> kernel: [ 1252.092406] cpu_startup_entry+0x73/0x75
Nov 18 19:38:24 <HOSTNAME> kernel: [ 1252.092412] rest_init+0xaa/0xac
Nov 18 19:38:24 <HOSTNAME> kernel: [ 1252.092418] start_kernel+0x540/0x561
Nov 18 19:38:24 <HOSTNAME> kernel: [ 1252.092420] ? set_init_arg+0x5a/0x5a
Nov 18 19:38:24 <HOSTNAME> kernel: [ 1252.092421] x86_64_start_reservations+0x32/0x34
Nov 18 19:38:24 <HOSTNAME> kernel: [ 1252.092423] x86_64_start_kernel+0x74/0x77
Nov 18 19:38:24 <HOSTNAME> kernel: [ 1252.092427] secondary_startup_64+0xa5/0xa5
Nov 18 19:38:24 <HOSTNAME> kernel: [ 1252.092429] Code: 00 a8 08 0f 84 67 ff ff ff eb bd 90 90 90 90 90 90 90 90 90 90 90 90 90 90 55 48 89 e5 e9 07 00 00 00 0f 00 2d 48 d9 59 00 fb f4 <5d> c3 66 90 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 e9 07 00
...
Nov 18 19:45:26 <HOSTNAME> kernel: [ 328.086724] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ora_p001_ipssg0:20681]
Nov 18 19:45:26 <HOSTNAME> kernel: [ 328.086762] Modules linked in: ovmapi sunrpc xfs libcrc32c ext4 mbcache jbd2 fscrypto dm_mod ppdev crct10dif_pclmul crc32_pclmul xen_fbfront ghash_clmulni_intel pcbc cirrus ttm drm_kms_helper aesni_intel crypto_simd glue_helper drm cryptd parport_pc pcspkr parport syscopyarea sysfillrect sysimgblt fb_sys_fops i2c_piix4 binfmt_misc ip_tables btrfs xor zstd_decompress zstd_compress xxhash raid6_pq xen_netfront ata_generic pata_acpi xen_blkfront ata_piix libata crc32c_intel serio_raw floppy
Nov 18 19:45:26 <HOSTNAME> kernel: [ 328.086793] CPU: 0 PID: 20681 Comm: ora_p001_ipssg0 Tainted: G L 4.14.35-1902.6.6.el7uek.x86_64 #2
Nov 18 19:45:26 <HOSTNAME> kernel: [ 328.086794] Hardware name: Xen HVM domU, BIOS 4.4.4OVM 11/16/2018
Nov 18 19:45:26 <HOSTNAME> kernel: [ 328.086795] task: ffff8facb01897c0 task.stack: ffffa02399dc4000
Nov 18 19:45:26 <HOSTNAME> kernel: [ 328.086802] RIP: 0010:mutex_lock+0xd/0x3a
Nov 18 19:45:26 <HOSTNAME> kernel: [ 328.086802] RSP: 0018:ffffa02399dc7da0 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff0c
Nov 18 19:45:26 <HOSTNAME> kernel: [ 328.086803] RAX: ffff8fb2f1068900 RBX: ffff8fb2f1068900 RCX: 00007fff0456c338
Nov 18 19:45:26 <HOSTNAME> kernel: [ 328.086804] RDX: 0000000000000080 RSI: 0000000000000003 RDI: ffff8fb2f1068900
Nov 18 19:45:26 <HOSTNAME> kernel: [ 328.086805] RBP: ffffa02399dc7da8 R08: ffffa02399dc7e38 R09: 00007fff0456c338
Nov 18 19:45:26 <HOSTNAME> kernel: [ 328.086806] R10: 0000000000000001 R11: 0000000000000080 R12: 0000000000000003
Nov 18 19:45:26 <HOSTNAME> kernel: [ 328.086806] R13: 0000000000000080 R14: 00007fff0456c338 R15: ffff8fb2f1068780
Nov 18 19:45:26 <HOSTNAME> kernel: [ 328.086808] FS: 00007f8c29477000(0000) GS:ffff8fba0d600000(0000) knlGS:0000000000000000
Nov 18 19:45:26 <HOSTNAME> kernel: [ 328.086809] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 18 19:45:26 <HOSTNAME> kernel: [ 328.086810] CR2: 00000000afc578b0 CR3: 00000031b0318005 CR4: 00000000001606f0
Nov 18 19:45:26 <HOSTNAME> kernel: [ 328.086813] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Nov 18 19:45:26 <HOSTNAME> kernel: [ 328.086813] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Nov 18 19:45:26 <HOSTNAME> kernel: [ 328.086814] Call Trace:
Nov 18 19:45:26 <HOSTNAME> kernel: [ 328.086820] aio_read_events+0x4d/0x2f4
Nov 18 19:45:26 <HOSTNAME> kernel: [ 328.086822] read_events+0xa0/0x209
Nov 18 19:45:26 <HOSTNAME> kernel: [ 328.086825] ? audit_filter_syscall+0x33/0xce
Nov 18 19:45:26 <HOSTNAME> kernel: [ 328.086827] ? __audit_syscall_entry+0xac/0xef
Nov 18 19:45:26 <HOSTNAME> kernel: [ 328.086831] ? syscall_trace_enter+0x1ce/0x2b8
Nov 18 19:45:26 <HOSTNAME> kernel: [ 328.086833] SyS_io_getevents+0x4d/0x9a
Nov 18 19:45:26 <HOSTNAME> kernel: [ 328.086835] do_syscall_64+0x79/0x1b9
Nov 18 19:45:26 <HOSTNAME> kernel: [ 328.086838] entry_SYSCALL_64_after_hwframe+0x169/0x0
Nov 18 19:45:26 <HOSTNAME> kernel: [ 328.086839] RIP: 0033:0x7f8c259bc644
Nov 18 19:45:26 <HOSTNAME> kernel: [ 328.086840] RSP: 002b:00007fff0456c320 EFLAGS: 00000297 ORIG_RAX: 00000000000000d0
Nov 18 19:45:26 <HOSTNAME> kernel: [ 328.086841] RAX: ffffffffffffffda RBX: 00007fff0456d338 RCX: 00007f8c259bc644
Nov 18 19:45:26 <HOSTNAME> kernel: [ 328.086842] RDX: 0000000000000080 RSI: 0000000000000003 RDI: 00007f8c29481000
Nov 18 19:45:26 <HOSTNAME> kernel: [ 328.086842] RBP: 00007fff0456d3e0 R08: 00007fff0456d338 R09: 00007fff0456c338
Nov 18 19:45:26 <HOSTNAME> kernel: [ 328.086843] R10: 00007fff0456c338 R11: 0000000000000297 R12: 00007f8c2943e920
Nov 18 19:45:26 <HOSTNAME> kernel: [ 328.086844] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000003
Nov 18 19:45:26 <HOSTNAME> kernel: [ 328.086845] Code: 44 00 00 55 be 02 00 00 00 48 89 e5 e8 ad f9 ff ff 5d c3 90 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 53 48 89 fb <e8> 0e e1 ff ff 65 48 8b 14 25 80 5c 01 00 31 c0 f0 48 0f b1 13

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


In this Document
Symptoms
Cause
Solution
References


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.