Server crashed unexpectedly with stack trace including: IP: [<ffffffff81350eb6>] sysrq_handle_crash+0x16/0x20 (Doc ID 2401543.1)

Last updated on JULY 11, 2023

Applies to:

Symptoms

A server which was part of a RAC cluster hung, and crashed, with a vmcore generated..

The vmcore shows the following:

KERNEL: /share/linuxrpm/vmlinux_repo/64/2.6.32-573.7.1.el6.x86_64/vmlinux
DUMPFILE: vmcore [PARTIAL DUMP]
CPUS: 48
DATE: Wed Jan 31 01:03:46 2018
UPTIME: 00:23:01
LOAD AVERAGE: 8.77, 3.10, 1.73
TASKS: 1995
NODENAME: server
RELEASE: 2.6.32-573.7.1.el6.x86_64
VERSION: #1 SMP Tue Sep 22 08:34:17 PDT 2015
MACHINE: x86_64 (2597 Mhz)
MEMORY: 383.9 GB
PANIC: "SysRq : Trigger a crash"
PID: 29990
COMMAND: "cssdagent"
TASK: ffff883030cb2040 [THREAD_INFO: ffff8817b1f18000]
CPU: 27
STATE: TASK_RUNNING (SYSRQ)

With the backtrace output showing the cssdagent triggered the crash via SysRq:

crash64> bt
PID: 29990 TASK: ffff883030cb2040 CPU: 27 COMMAND: "cssdagent"
#0 [ffff8817b1f1b9e0] machine_kexec at ffffffff8103d1ab
#1 [ffff8817b1f1ba40] crash_kexec at ffffffff810cc4a2
#2 [ffff8817b1f1bb10] oops_end at ffffffff8153cc00
#3 [ffff8817b1f1bb40] no_context at ffffffff8104e8cb
#4 [ffff8817b1f1bb90] __bad_area_nosemaphore at ffffffff8104eb55
#5 [ffff8817b1f1bbe0] bad_area at ffffffff8104ec7e
#6 [ffff8817b1f1bc10] __do_page_fault at ffffffff8104f483
#7 [ffff8817b1f1bd30] do_page_fault at ffffffff8153eb4e
#8 [ffff8817b1f1bd60] page_fault at ffffffff8153bef5
[exception RIP: sysrq_handle_crash+22]
RIP: ffffffff81350eb6 RSP: ffff8817b1f1be18 RFLAGS: 00010096
RAX: 0000000000000010 RBX: 0000000000000063 RCX: 000000000000c730
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000063
RBP: ffff8817b1f1be18 R8: 0000000000000000 R9: 0000000000000001
R10: ffffffff8164b260 R11: ffff88603383a65f R12: 0000000000000000
R13: ffffffff81b109a0 R14: 0000000000000286 R15: 0000000000000004
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
#9 [ffff8817b1f1be20] __handle_sysrq at ffffffff81351172
#10 [ffff8817b1f1be70] write_sysrq_trigger at ffffffff8135122e
#11 [ffff8817b1f1bea0] proc_reg_write at ffffffff811fd27e
#12 [ffff8817b1f1bef0] vfs_write at ffffffff81191a98
#13 [ffff8817b1f1bf30] sys_write at ffffffff811925d1
#14 [ffff8817b1f1bf80] system_call_fastpath at ffffffff8100b0d2
RIP: 00000034a460e77d RSP: 00007f0523ffdb28 RFLAGS: 00000246
RAX: 0000000000000001 RBX: ffffffff8100b0d2 RCX: 00000034a460ba0e
RDX: 0000000000000001 RSI: 00007f053633e54c RDI: 0000000000000004
RBP: 00007f0523ffdb30 R8: 0000000000000058 R9: 0000000000140000
R10: 00007f0523ffd8c0 R11: 0000000000000293 R12: 00007f0523fff9c0
R13: 00007f0531e65870 R14: 0000000000006cc0 R15: 00007f0531e65870
ORIG_RAX: 0000000000000001 CS: 0033 SS: 002b

Cause

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.