Storage cell in reboot cycle due to kernel panic (Doc ID 1541148.1)

Last updated on APRIL 02, 2013

Applies to:

Oracle Exadata Storage Server Software - Version 11.1.0.3.0 to 11.2.3.2.1 [Release 11.1 to 11.2]
Linux x86-64

Symptoms

Storage cell is stuck in a reboot loop initiated by the kernel hitting a kernel panic due to Unable to handle kernel NULL pointer dereference at some point during every boot. Call traces similar to the following may be seen over the serial console, ILOM Remote Console or captured by the ILOM in the serial console logs: ILOM snapshot's ilom\@persist@hostconsole.log or @persist@hostconsole.log.1.

While call stack and place of panic usually varies, the key symptom is the fact that it also occurs when booting the GRUB entry CELL_USB_BOOT_CELLBOOT_usb_in_rescue_mode or when booting from diagnostics.iso. Another key symptom, which can be observed in the serial console log, is a message that starts with Booting processor, occurring right before the PANIC.

 

Excerpt from ilom\@persist@hostconsole.log:

Intel(R) Xeon(R) CPU           L5640  @ 2.27GHz stepping 02
SMP alternatives: switching to SMP code
Booting processor 6/12 APIC 0x1
Unable to handle kernel NULL pointer dereference at 0000000000000008 RIP:
 [<ffffffff80097b37>] run_timer_softirq+0xe3/0x241
PGD 0
Oops: 0002 [1] SMP
last sysfs file:
CPU 3
Modules linked in:
Pid: 0, comm: swapper Not tainted 2.6.18-194.3.1.0.4.el5 #1
RIP: 0010:[<ffffffff80097b37>]  [<ffffffff80097b37>] run_timer_softirq+0xe3/0x241
RSP: 0018:ffff81010c5bff10  EFLAGS: 00010006
RAX: ffff81010c5d0060 RBX: 0000000000002006 RCX: ffff81010c5d0078
RDX: 0000000000000000 RSI: ffff81010c5bfee0 RDI: ffff81010c5b0000
RBP: ffff81010c5bff10 R08: 0000000000000003 R09: ffff81010c5b9e48
R10: 0000000000000001 R11: ffff81010c5bfee0 R12: ffff81010c5b0000
R13: 000000000000000a R14: 0000000000000003 R15: ffffffff803f42c0
FS:  0000000000000000(0000) GS:ffff81010c5636c0(0000) knlGS:0000000000000000
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 0000000000000008 CR3: 0000000000201000 CR4: 00000000000006a0
Process swapper (pid: 0, threadinfo ffff81010c5b8000, task ffff81010c570100)
Stack:  0000000000000000 ffff81010c5bff10 ffff81010c570100 0000000000000046
 0000000000000001 ffffffff803e1f90 000000000000000a 0000000000000003
 ffffffff803f42c0 ffffffff800123da 0000000000000046 ffff81010c5bff98
Call Trace:
 <IRQ>  [<ffffffff800123da>] __do_softirq+0x89/0x133
 [<ffffffff8005e2fc>] call_softirq+0x1c/0x28
 [<ffffffff8006cbaa>] do_softirq+0x2c/0x85
 [<ffffffff80057287>] mwait_idle+0x0/0x4a
 [<ffffffff8005dc8e>] apic_timer_interrupt+0x66/0x6c
 <EOI>  [<ffffffff800572bd>] mwait_idle+0x36/0x4a
 [<ffffffff80049475>] cpu_idle+0x95/0xb8
 [<ffffffff80077996>] start_secondary+0x498/0x4a7


Code: 48 89 62 08 48 8b 51 08 48 89 54 24 08 48 89 22 48 89 49 08
RIP  [<ffffffff80097b37>] run_timer_softirq+0xe3/0x241
 RSP <ffff81010c5bff10>
CR2: 0000000000000008
 <0>Kernel panic - not syncing: Fatal exception
 <0>Rebooting in 60 seconds..

 

Screenshot of ILOM Remote Console

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms