Kernel zone running Solaris 11.4 hangs/panics in xcall_serv()
(Doc ID 2864188.1)
Last updated on APRIL 20, 2022
Applies to:
Solaris Operating System - Version 11.4 to 11.4 [Release 11.0]Information in this document applies to any platform.
Symptoms
In this case the kernel zone was hung and a live dump was captured.
Reviewing the live kernel corefile with SCAT shows:
1. The WARNINGS report cpu3 running a thread for nearly 22 hours.
WARNING: last_swtch[3]: 0x1cca5fa0ecebf1 (21 hours 59 minutes 11.916633592 seconds earlier) <----
WARNING: SYS thread 0xffffe33001035ac0 on CPU3 using 100%CPU
WARNING: 4 CPUs have 390 threads in their dispatch queues
2. Checking cpu3 shows there are many other threads back up behind the one still running on the cpu.
CAT(kzcore.0/11X)> cpu
CPU thread pri PID cmd
...
3 @ 0xffffe08000600000 0xffffe33001035ac0 99 sched(callout_normal_taskq) <----
pri 60 -=> 0xffffe330010b0ac0 60 sched(genunix:seg_pasync_thread)
0xffffe330002a5ac0 60 sched(zfs:l2arc_feed_thread)
0xffffe3300001dac0 60 sched(genunix:thread_reaper)
...
3. Examining the thread running on the cpu. Note again, this is a live dump.
CAT(kzcore.0/11X)> thread 0xffffe33001035ac0
==== kernel thread: 0xffffe33001035ac0 PID: 0 on CPU: 3 affinity CPU: 3 ====
cmd: sched(callout_normal_taskq)
kname: callout_normal_taskq
t_procp: 0xfffffffffc081ef0 (proc_sched)
p_as: 0xfffffffffc085490 (kas)
p_zone: 0xfffffffffc2c97c0 (global)
t_stk: 0xffffe33001035ac0 sp: 0xffffe33000ecc9e8 t_stkbase: 0xffffe33001031000
t_pri: 99 (SYS) pctcpu: 100.000000
t_transience: 1
t_cpupart: 0xfffffffffc0fef90(0) last CPU: 3
idle: 79171619617873 hrticks (21h59m31.619617873s)
start: Sat Feb 19 10:55:24 2022
age: 3203261 seconds (37 days 1 hours 47 minutes 41 seconds)
t_state: TS_ONPROC
t_flag: 0x10808(T_TALLOCSTK|T_PANIC|T_PUSHPAGE)
t_proc_flag: 0 (none set)
t_schedflag: 0x13 (TS_LOAD|TS_DONT_SWAP|TS_SIGNALLED)
t_acflag: 4 (TA_NO_ACCOUNTING)
p_flag: 1 (SSYS)
pc: unix:panicsys+0x90: leaq -0x80(%rbp),%rbx
void unix:panicsys+0x90((const char *)0xfffffffffb9a1a10, (struct __va_list_element *)0xffffe33000eccc00, (struct regs *)0xffffe33000ecc9f8, (int)0)
unix:vpanic_common+0x13a()
unix:0xfffffffffb8c4bfa()
void unix:die+0x142((uint_t)0xe, (struct regs *)0xffffe33000eccdc0, (caddr_t)0, (processorid_t)3)
void unix:trap+0x1437((struct regs *)0xffffe33000eccdc0, (caddr_t)0, (processorid_t)3)
-- panic trap data type: 0xe (Page fault)
...
void unix:panic_stopcpus+0x26((cpu_t *)0xffffe08000600000, (kthread_t *)0xffffe33001035ac0, (int)0xf)
void unix:panicsys+0x5bf((const char *)0xfffffffffb9a2540, (struct __va_list_element *)0xffffe33001035040, (struct regs *)0xfffffffffc0fc770, (int)1)
unix:vpanic_common+0x13a()
unix:0xfffffffffb8c4bfa()
unix:0xfffffffffb873836()
hrtime_t unix:xc_serv_delay+0x84((ulong_t *)0xffffe330010351f0, (hrtime_t)0x74de76ccf2, (hrtime_t)0x2f9b80)
uint_t unix:xc_serv+0x1f8((caddr_t)0xffffe330010351f0, (caddr_t)0)
void unix:xc_common+0x291((xc_func_t)0xfffffffffb840e00, (xc_arg_t)0xffffa10001f2b5a8, (xc_arg_t)0xffffe33001035520, (xc_arg_t)0, (ulong_t *)0xffffe330010353c0, (uint_t)2)
void unix:xc_call+0x46((xc_arg_t)0xffffa10001f2b5a8, (xc_arg_t)0xffffe33001035520, (xc_arg_t)0, (ulong_t *)0xffffe330010353c0, (xc_func_t)0xfffffffffb840e00)
void unix:hat_tlb_inval+0x4b3((hat_t *)0xffffa10001f2b5a8, (demap_range_t *)0xffffe33001035520)
x86pte_t unix:x86pte_inval+0xe4((htable_t *)0xffffa100020b9570, (uint_t)0x102, (x86pte_t)0x8000000103e65461, (x86pte_t *)0, (boolean_t)1)
void unix:hat_pte_unmap+0xe5((htable_t *)0xffffa100020b9570, (uint_t)0x102, (uint_t)4, (x86pte_t)0x8000000103e65461, (void *)0, (boolean_t)1)
void unix:hat_unload_callback+0x419((hat_t *)0xffffa10001f2b5a8, (caddr_t)0xffffa10002b02000, (size_t)0x1000, (uint_t)4, (hat_callback_t *)0)
void unix:hat_unload+0x33((hat_t *)0xffffa10001f2b5a8, (caddr_t)0xffffa10002b02000, (size_t)0x1000, (uint_t)4)
...
Cause
To view full details, sign in with your My Oracle Support account. |
|
Don't have a My Oracle Support account? Click to get started! |
In this Document
| Symptoms |
| Cause |
| Solution |