My Oracle Support Banner

Kernel zone running Solaris 11.4 hangs/panics in xcall_serv() (Doc ID 2864188.1)

Last updated on APRIL 20, 2022

Applies to:

Solaris Operating System - Version 11.4 to 11.4 [Release 11.0]
Information in this document applies to any platform.


In this case the kernel zone was hung and a live dump was captured.

Reviewing the live kernel corefile with SCAT shows:

1. The WARNINGS report cpu3 running a thread for nearly 22 hours.

     WARNING: last_swtch[3]: 0x1cca5fa0ecebf1 (21 hours 59 minutes 11.916633592 seconds earlier) <----
     WARNING: SYS thread 0xffffe33001035ac0 on CPU3 using 100%CPU
     WARNING: 4 CPUs have 390 threads in their dispatch queues


2. Checking cpu3 shows there are many other threads back up behind the one still running on the cpu.

     CAT(kzcore.0/11X)> cpu
     CPU thread pri PID cmd
     3 @ 0xffffe08000600000 0xffffe33001035ac0 99 sched(callout_normal_taskq) <----
     pri 60 -=> 0xffffe330010b0ac0 60 sched(genunix:seg_pasync_thread)
     0xffffe330002a5ac0 60 sched(zfs:l2arc_feed_thread)
     0xffffe3300001dac0 60 sched(genunix:thread_reaper)


3. Examining the thread running on the cpu. Note again, this is a live dump.

     CAT(kzcore.0/11X)> thread 0xffffe33001035ac0
     ==== kernel thread: 0xffffe33001035ac0 PID: 0 on CPU: 3 affinity CPU: 3 ====
     cmd: sched(callout_normal_taskq)
     kname: callout_normal_taskq
     t_procp: 0xfffffffffc081ef0 (proc_sched)
     p_as: 0xfffffffffc085490 (kas)
     p_zone: 0xfffffffffc2c97c0 (global)
     t_stk: 0xffffe33001035ac0 sp: 0xffffe33000ecc9e8 t_stkbase: 0xffffe33001031000
     t_pri: 99 (SYS) pctcpu: 100.000000
     t_transience: 1
     t_cpupart: 0xfffffffffc0fef90(0) last CPU: 3
     idle: 79171619617873 hrticks (21h59m31.619617873s)
     start: Sat Feb 19 10:55:24 2022
     age: 3203261 seconds (37 days 1 hours 47 minutes 41 seconds)
     t_state: TS_ONPROC
     t_flag: 0x10808(T_TALLOCSTK|T_PANIC|T_PUSHPAGE)
     t_proc_flag: 0 (none set)
     t_schedflag: 0x13 (TS_LOAD|TS_DONT_SWAP|TS_SIGNALLED)
     t_acflag: 4 (TA_NO_ACCOUNTING)
     p_flag: 1 (SSYS)

     pc: unix:panicsys+0x90: leaq -0x80(%rbp),%rbx

     void unix:panicsys+0x90((const char *)0xfffffffffb9a1a10, (struct __va_list_element *)0xffffe33000eccc00, (struct regs *)0xffffe33000ecc9f8, (int)0)
     void unix:die+0x142((uint_t)0xe, (struct regs *)0xffffe33000eccdc0, (caddr_t)0, (processorid_t)3)
     void unix:trap+0x1437((struct regs *)0xffffe33000eccdc0, (caddr_t)0, (processorid_t)3)
     -- panic trap data type: 0xe (Page fault)
     void unix:panic_stopcpus+0x26((cpu_t *)0xffffe08000600000, (kthread_t *)0xffffe33001035ac0, (int)0xf)
     void unix:panicsys+0x5bf((const char *)0xfffffffffb9a2540, (struct __va_list_element *)0xffffe33001035040, (struct regs *)0xfffffffffc0fc770, (int)1)
     hrtime_t unix:xc_serv_delay+0x84((ulong_t *)0xffffe330010351f0, (hrtime_t)0x74de76ccf2, (hrtime_t)0x2f9b80)
     uint_t unix:xc_serv+0x1f8((caddr_t)0xffffe330010351f0, (caddr_t)0)
     void unix:xc_common+0x291((xc_func_t)0xfffffffffb840e00, (xc_arg_t)0xffffa10001f2b5a8, (xc_arg_t)0xffffe33001035520, (xc_arg_t)0, (ulong_t *)0xffffe330010353c0, (uint_t)2)
     void unix:xc_call+0x46((xc_arg_t)0xffffa10001f2b5a8, (xc_arg_t)0xffffe33001035520, (xc_arg_t)0, (ulong_t *)0xffffe330010353c0, (xc_func_t)0xfffffffffb840e00)
     void unix:hat_tlb_inval+0x4b3((hat_t *)0xffffa10001f2b5a8, (demap_range_t *)0xffffe33001035520)
     x86pte_t unix:x86pte_inval+0xe4((htable_t *)0xffffa100020b9570, (uint_t)0x102, (x86pte_t)0x8000000103e65461, (x86pte_t *)0, (boolean_t)1)
     void unix:hat_pte_unmap+0xe5((htable_t *)0xffffa100020b9570, (uint_t)0x102, (uint_t)4, (x86pte_t)0x8000000103e65461, (void *)0, (boolean_t)1)
     void unix:hat_unload_callback+0x419((hat_t *)0xffffa10001f2b5a8, (caddr_t)0xffffa10002b02000, (size_t)0x1000, (uint_t)4, (hat_callback_t *)0)
     void unix:hat_unload+0x33((hat_t *)0xffffa10001f2b5a8, (caddr_t)0xffffa10002b02000, (size_t)0x1000, (uint_t)4)



To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!

In this Document

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.