Solaris 10 guest ldom can hang or reboot frequently due to hundreds of expired callouts (Doc ID 2539071.1)

Last updated on MAY 17, 2023

Applies to:

Symptoms

SPARC servers with M8/T8 CPU installed and running Solaris 10 frequently rebooted or hang.

Crash dump shows hundreds of expired callouts.
CAT(vmcore.0/10V)> callout -e
CPU RHLEX XID expiration function(arg)
=== ===== ================== ======================== =============
2 R LE 0x24598c7404 -14h6m54.909080820s genunix:cv_wakeup((void *)0x2a100197c80)
2 R LE 0x245ae57404 -14h6m51.598565030s genunix:cv_wakeup((void *)0x300085dfbc0)
2 R LE 0x245999f004 -14h6m54.249080820s genunix:clock_realtime_fire((void *)0x60029dcde50)
2 R LE 0x24598c7004 -14h6m52.282332965s genunix:cv_wakeup((void *)0x30008550080)
2 R LE 0x24598c6c04 -14h6m50.689080820s genunix:lwp_timer_timeout((void *)0x2a101265a20)
2 R LE 0x24598c6404 -14h6m44.689080820s genunix:lwp_timer_timeout((void *)0x2a100bb9a20)
2 R LE 0x24598c8404 -14h6m49.959080820s genunix:cv_wakeup((void *)0x2a100ec7c80)
2 R LE 0x245ae37804 -14h6m48.493283065s genunix:cv_wakeup((void *)0x300085f4b40)
2 R LE 0x245bb64804 -14h6m17.679080820s genunix:sigalarm2proc((void *)0x6002b576d08)
2 R LE 0x245ae4e404 -14h5m31.013279865s genunix:cv_wakeup((void *)0x3000ae2d200)
2 R LE 0x245bc99804 -14h5m21.689080820s genunix:sigalarm2proc((void *)0x600246c8010)
2 R LE 0x245979b404 -14h4m59.353613650s genunix:cv_wakeup((void *)0x30008612040)

...

Notice in this instance the expired callouts were queued for CPU2.

Looking at CPU2 we see that it is running its idle() thread:

CAT(vmcore.0/10V)> cpu 2
CPU thread pri PID cmd
2 @ 0x30007d8a000 0x2a1003a5c80 -1 sched(unix:idle) (idle)

We also see that CPU2 has a base SPL of 0 :

CAT(vmcore.0/10V)> sdump 0x30007d8a000 cpu cpu_base_spl
int cpu_base_spl = 0

Cause

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.