Exadata DB Node Panic With 3rd Party Software (Doc ID 1397364.1)

Last updated on SEPTEMBER 28, 2021

Applies to:

Oracle Exadata Storage Server Software - Version 11.2.2.2.0 to 11.2.3.2.0 [Release 11.2]
Information in this document applies to any platform.

Symptoms

Database node panics and a crash core has been created with a stack similar to below.
Analyzing the core using crash command:

KERNEL: /usr/lib/debug/lib/modules/2.6.18-194.3.1.0.4.el5/vmlinux
   DUMPFILE: crashcore [PARTIAL DUMP]
       CPUS: 24
       DATE: Wed May 9 04:22:36 2012
     UPTIME: 01:14:26
LOAD AVERAGE: 0.08, 0.05, 0.08
      TASKS: 468
   NODENAME: <HOSTNAME>.<DOMAINNAME>.com
    RELEASE: 2.6.18-194.3.1.0.4.el5
    VERSION: #1 SMP Sat Feb 19 03:38:37 EST 2011
    MACHINE: x86_64 (2926 Mhz)
     MEMORY: 94.6 GB
      PANIC: "Oops: 0002 [1] SMP " (check log for details)
        PID: 5293
    COMMAND: "bpbkar"
       TASK: ffff81187da00860 [THREAD_INFO: ffff811862e58000]
        CPU: 16
      STATE: TASK_RUNNING (PANIC)

Dumping the callstack of the process causing the crash:

crash> bt
PID: 5293   TASK: ffff81187da00860 CPU: 16 COMMAND: "bpbkar"
#0 [ffff811862e59a30] crash_kexec at ffffffff800adadc
#1 [ffff811862e59af0] __die at ffffffff80065173
#2 [ffff811862e59b30] do_page_fault at ffffffff80066df3
#3 [ffff811862e59c20] error_exit at ffffffff8005dde9
    [exception RIP: number+420]
    RIP: ffffffff800103df RSP: ffff811862e59cd8 RFLAGS: 00010293
    RAX: 0000000000000034 RBX: 000000000000000a RCX: 000000000000000a
    RDX: 00000000fffffff3 RSI: 000000007fffffff RDI: 0000000000000000
    RBP: 0000000000000000   R8: 00000000ffffffff   R9: 0000000000000020
    R10: 0000000000000009 R11: 0000000000000000 R12: ffffffff80294120
    R13: 00000000ffffffff R14: 0000000000000000 R15: 000000000000000a
    ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
#4 [ffff811862e59d60] vsnprintf at ffffffff8001a7cf
#5 [ffff811862e59eb8] audit_filter_syscall at ffffffff800b698d
#6 [ffff811862e59f10] vfs_read at ffffffff8000b681
#7 [ffff811862e59f40] sys_read at ffffffff80011bd2
#8 [ffff811862e59f80] sysenter_do_call at ffffffff8006149d
    RIP: 00000000ffffe410 RSP: 00000000ff984a18 RFLAGS: 00000296
    RAX: ffffffffffffffda RBX: ffffffff8006149d RCX: 00000000094ba720
    RDX: 0000000000010000 RSI: 0000000000000000 RDI: 0000000000000000
    RBP: 0000000000000000   R8: 0000000000000000   R9: 0000000000000000
    R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
    R13: 00000000ff984ac8 R14: 0000000000000012 R15: 0000000000000000
    ORIG_RAX: 0000000000000003 CS: 0023 SS: 002b

Listing the files opened by the process:

PID: 5293   TASK: ffff81187da00860 CPU: 16 COMMAND: "bpbkar"
ROOT: /    CWD: /sys/class/scsi_host/host0
FD       FILE            DENTRY           INODE       TYPE PATH
0 ffff8111cf61b280 ffff81113b23ca98 ffff81186f1ded50 SOCK socket:/[1154382]
1 ffff810b085eb880 ffff81113b23c228 ffff81186f1de0d0 SOCK socket:/[1154374]
2 ffff810dfdf393c0 ffff81113b23c810 ffff81186f1dead0 SOCK socket:/[1154475]
3 ffff81187f5c1580 ffff81187e216588 ffff81187da11d30 DIR /
4 ffff810afb17ca80 ffff810155cb3d20 ffff810155cb1d70 DIR /sys/
5 ffff81187d89ba80 ffff81187f505b70 ffff81187f504910 DIR /sys/class
6 ffff811274dad6c0 ffff81187fe6c660 ffff81187df4d910 DIR /sys/class/scsi_host
7 ffff811274dad0c0 ffff81187e230df8 ffff81187ec18910 DIR /sys/class/scsi_host/host0
8 ffff8114ce6cbdc0 ffff8117c6d38738 ffff811816004280 REG /sys/class/net/bond0/carrier
9 ffff8114ce6cb6c0 ffff8117c6d38810 ffff8118160044b0 REG /sys/class/net/bond0/dormant
10 ffff810afb17c980 ffff811225ba14b0 ffff810c02bc4280 REG /sys/class/net/bondeth1/carrier
11 ffff81187d89b580 ffff811225ba1588 ffff810c02bc44b0 REG /sys/class/net/bondeth1/dormant
12 ffff81187f984080 ffff810bfdbdda98 ffff810af4aa2910 REG /sys/class/net/eth4/carrier
13 ffff8111cf61bc80 ffff810bfdbddb70 ffff810af4aa2b40 REG /sys/class/net/eth4/dormant
14 ffff81187979cac0 ffff811806e2f8e8 ffff81153b879280 REG /sys/class/net/eth5/carrier
15 ffff81187979c0c0 ffff811806e2f9c0 ffff81153b8794b0 REG /sys/class/net/eth5/dormant
16 ffff810c203ebe80 ffff811467b90588 ffff810b215826e0 REG /sys/class/net/sit0/carrier
17 ffff81187de9c480 ffff811467b90660 ffff810b21582910 REG /sys/class/net/sit0/dormant
18 ffff811825bd65c0 ffff810c7d483df8 ffff810c38410910 REG /sys/class/scsi_host/host0/max_sectors

Note that files opened by the process are OS related files located under /sys.

Changes

3rd party backup software was performing system backup while the system panics.

Cause

	To view full details, sign in with your My Oracle Support account.
	Don't have a My Oracle Support account? Click to get started!

In this Document

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.

Exadata DB Node Panic With 3rd Party Software (Doc ID 1397364.1)

Applies to:

Symptoms

Changes

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!