Guest VM Got Stuck In Reboot Process in Exadata Cloud During VM OS Patching
(Doc ID 3047559.1)
Last updated on SEPTEMBER 13, 2024
Applies to:
Gen 2 Exadata Cloud at Customer - Version N/A and laterOracle Cloud Infrastructure - Exadata Cloud Service - Version N/A and later
Linux x86-64
Symptoms
On Exadata Cloud while running VM OS Patch to 23.1.16 and 23.1.17, VM can stop responding or get stuck during the reboot step.
While checking the VM we could connect however imageinfo command is hanging.
In the OS Messages we can observe kernel:err: like followings
2024-09-09T19:30:59.890688-04:00 hostname kernel:err: INFO: task kworker/2:0:23 blocked for more than 122 seconds. <<<<<<<
2024-09-09T19:30:59.890825-04:00 hostname kernel:err: Tainted: G O 5.4.17-2136.322.6.5.el8uek.x86_64 #2
2024-09-09T19:30:59.890843-04:00 hostname kernel:err: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
2024-09-09T19:30:59.893140-04:00 hostname kernel:info: kworker/2:0 D 0 23 2 0x80004080 <<<<<<<
2024-09-09T19:30:59.893167-04:00 hostname kernel:info: Workqueue: events netstamp_clear
2024-09-09T19:30:59.893188-04:00 hostname kernel:warning: Call Trace:
2024-09-09T19:30:59.893214-04:00 hostname kernel:warning: __schedule+0x29f/0x5f7
2024-09-09T19:30:59.893236-04:00 hostname kernel:warning: ? __update_load_avg_se+0x232/0x324
2024-09-09T19:30:59.893250-04:00 hostname kernel:warning: schedule+0x44/0xb3
2024-09-09T19:30:59.893269-04:00 hostname kernel:warning: schedule_preempt_disabled+0xe/0x14
2024-09-09T19:30:59.893283-04:00 hostname kernel:warning: __mutex_lock.isra.8+0x287/0x465
2024-09-09T19:30:59.893302-04:00 hostname kernel:warning: __mutex_lock_slowpath+0x13/0x19
2024-09-09T19:30:59.893315-04:00 hostname kernel:warning: mutex_lock+0x2c/0x33
2024-09-09T19:30:59.893329-04:00 hostname kernel:warning: arch_jump_label_transform_apply+0x24/0x52
2024-09-09T19:30:59.893343-04:00 hostname kernel:warning: __jump_label_update+0x116/0x125
2024-09-09T19:30:59.893358-04:00 hostname kernel:warning: jump_label_update+0xaa/0xd2
2024-09-09T19:30:59.893372-04:00 hostname kernel:warning: static_key_enable_cpuslocked+0x62/0x88
2024-09-09T19:30:59.893386-04:00 hostname kernel:warning: static_key_enable+0x1a/0x26
2024-09-09T19:30:59.893400-04:00 hostname kernel:warning: netstamp_clear+0x2d/0x3e
2024-09-09T19:30:59.893413-04:00 hostname kernel:warning: process_one_work+0x1bb/0x3a9
2024-09-09T19:30:59.893434-04:00 hostname kernel:warning: worker_thread+0x37/0x3b2
2024-09-09T19:30:59.893448-04:00 hostname kernel:warning: kthread+0x120/0x136
2024-09-09T19:30:59.893464-04:00 hostname kernel:warning: ? create_worker+0x1b0/0x1ab
2024-09-09T19:30:59.893478-04:00 hostname kernel:warning: ? __kthread_cancel_work+0x50/0x46
2024-09-09T19:30:59.893494-04:00 hostname kernel:warning: ret_from_fork+0x24/0x36
2024-09-09T19:30:59.893511-04:00 hostname kernel:err: INFO: task wdavdaemon:15638 blocked for more than 122 seconds. <<<<<<<
2024-09-09T19:30:59.893529-04:00 hostname kernel:err: Tainted: G O 5.4.17-2136.322.6.5.el8uek.x86_64 #2
2024-09-09T19:30:59.895580-04:00 hostname kernel:err: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
2024-09-09T19:30:59.895600-04:00 hostname kernel:info: wdavdaemon D 0 15638 1 0x00004080 <<<<<<<
2024-09-09T19:30:59.895619-04:00 hostname kernel:warning: Call Trace:
2024-09-09T19:30:59.895634-04:00 hostname kernel:warning: __schedule+0x29f/0x5f7
2024-09-09T19:30:59.895650-04:00 hostname kernel:warning: schedule+0x44/0xb3
2024-09-09T19:30:59.895690-04:00 hostname kernel:warning: schedule_preempt_disabled+0xe/0x14
2024-09-09T19:30:59.895704-04:00 hostname kernel:warning: __mutex_lock.isra.8+0x287/0x465
2024-09-09T19:30:59.895718-04:00 hostname kernel:warning: __mutex_lock_slowpath+0x13/0x19
2024-09-09T19:30:59.895731-04:00 hostname kernel:warning: mutex_lock+0x2c/0x33
2024-09-09T19:30:59.895751-04:00 hostname kernel:warning: set_all_modules_text_rw+0x26/0x96
2024-09-09T19:30:59.895769-04:00 hostname kernel:warning: ftrace_arch_code_modify_prepare+0x1a/0x22
2024-09-09T19:30:59.895783-04:00 hostname kernel:warning: ftrace_run_update_code+0xc/0x6d
2024-09-09T19:30:59.895799-04:00 hostname kernel:warning: ftrace_hash_move_and_update_ops+0x1ce/0x23f
2024-09-09T19:30:59.895812-04:00 hostname kernel:warning: ftrace_set_hash+0x123/0x1c7
2024-09-09T19:30:59.895829-04:00 hostname kernel:warning: ? __x64_sys_unlink+0x30/0x29
2024-09-09T19:30:59.895843-04:00 hostname kernel:warning: ftrace_set_filter_ip+0x2e/0x6f
2024-09-09T19:30:59.895857-04:00 hostname kernel:warning: arm_kprobe+0xac/0x14b
2024-09-09T19:30:59.895935-04:00 hostname kernel:warning: enable_kprobe+0x86/0xa6
2024-09-09T19:30:59.895951-04:00 hostname kernel:warning: enable_trace_kprobe+0xa5/0x148
2024-09-09T19:30:59.895965-04:00 hostname kernel:warning: kprobe_register+0x3b/0x4f
2024-09-09T19:30:59.895999-04:00 hostname kernel:warning: perf_kprobe_init+0xcb/0xfa
2024-09-09T19:30:59.896012-04:00 hostname kernel:warning: perf_kprobe_event_init+0x3e/0x6c
2024-09-09T19:30:59.896025-04:00 hostname kernel:warning: perf_try_init_event+0x48/0x12c
2024-09-09T19:30:59.896038-04:00 hostname kernel:warning: perf_event_alloc+0x866/0xc80 [ksplice_hyxcmez5_vmlinux_new]
2024-09-09T19:30:59.896053-04:00 hostname kernel:warning: __do_sys_perf_event_open+0x1a2/0xe89
2024-09-09T19:30:59.896066-04:00 hostname kernel:warning: __x64_sys_perf_event_open+0x20/0x26
2024-09-09T19:30:59.896082-04:00 hostname kernel:warning: do_syscall_64+0x60/0x1cf
2024-09-09T19:30:59.896099-04:00 hostname kernel:warning: entry_SYSCALL_64_after_hwframe+0x175/0x0
2024-09-09T19:30:59.896112-04:00 hostname kernel:warning: RIP: 0033:0x7f9a6ebe941d
2024-09-09T19:30:59.896126-04:00 hostname kernel:warning: Code: Bad RIP value.
2024-09-09T19:30:59.896152-04:00 hostname kernel:warning: RSP: 002b:00007f9a3d1894a8 EFLAGS: 00000246 ORIG_RAX: 000000000000012a
2024-09-09T19:30:59.896166-04:00 hostname kernel:warning: RAX: ffffffffffffffda RBX: 0000000000000006 RCX: 00007f9a6ebe941d
2024-09-09T19:30:59.896179-04:00 hostname kernel:warning: RDX: 0000000000000000 RSI: 00000000ffffffff RDI: 00007f9a3d1894c0
2024-09-09T19:30:59.896198-04:00 hostname kernel:warning: RBP: 0000000000000000 R08: 0000000000000008 R09: 00007f9a6430cda5
2024-09-09T19:30:59.896214-04:00 hostname kernel:warning: R10: 00000000ffffffff R11: 0000000000000246 R12: 00007f9a6430cda5
2024-09-09T19:30:59.896230-04:00 hostname kernel:warning: R13: 0000000000000000 R14: 0000000000000000 R15: 00000000ffffffff2024-09-09T19:30:59.895985-04:00 hostname kernel:warning: perf_trace_event_init+0x176/0x26e
wdavdaemon in this case is a process that runs in the background when Windows Defender is installed.
we know this same issue could help also with oneagenteventst which is a process that runs in the background when Dynatrace OneAgent is installed.
Changes
Customer has third party security agent installed on guests VMs.
Cause
To view full details, sign in with your My Oracle Support account. |
|
Don't have a My Oracle Support account? Click to get started! |
In this Document
Symptoms |
Changes |
Cause |
Solution |
References |