My Oracle Support Banner

Exadata server crashed WITH "BUG: unable to handle kernel paging request" (Doc ID 2583832.1)

Last updated on SEPTEMBER 03, 2019

Applies to:

Linux OS - Version Oracle Linux 6.1 and later
Exadata X5-2 Hardware - Version All Versions to All Versions [Release All Releases]
Information in this document applies to any platform.

Symptoms

A kernel crash has caused the system to reboot.

On : Exadata Image kernel version: 4.1.12-94.7.8.el6uek
Kernel version: 18.1.4.0.0.180125.3

Oracle Linux Server 6.9

Server Model: Oracle Corporation ORACLE SERVER X5-2

Exdata server crashed with following logs in vmcore.

KERNEL:
/share/linuxrpm/vmlinux_repo/64/nano/4.1.12-94.7.8.el6uek.x86_64/vmlinux
DUMPFILE: xxxxxxxx.xxx.xxxxxxxx.xxx.xx.vmcore [PARTIAL DUMP]
CPUS: 72
DATE: Wed Mar 6 04:09:45 2019
UPTIME: 238 days, 05:49:19
LOAD AVERAGE: 31.60, 31.69, 31.12
TASKS: 6974
NODENAME: xxxxxxxx.xxx.xxxxxxxx.xxx.xx
RELEASE: 4.1.12-94.7.8.el6uek.x86_64
VERSION: #2 SMP Thu Jan 11 20:41:01 PST 2018
MACHINE: x86_64 (2294 Mhz)
MEMORY: 511.9 GB
PANIC: "BUG: unable to handle kernel paging request at
ffff885020478000"
PID: 342933
COMMAND: "oracle"
TASK: ffff88418dc40000 [THREAD_INFO: ffff884187a00000]
CPU: 67
STATE: TASK_RUNNING (PANIC)

> bt
PID: 342933 TASK: ffff88418dc40000 CPU: 67 COMMAND: "oracle"
#0 [ffff884187a037b0] machine_kexec at ffffffff8105eb10
#1 [ffff884187a03820] crash_kexec at ffffffff81114748
#2 [ffff884187a038f0] oops_end at ffffffff8101a848
#3 [ffff884187a03920] no_context at ffffffff8106ea51
#4 [ffff884187a03970] __bad_area_nosemaphore at ffffffff8106ec4d
#5 [ffff884187a039c0] bad_area_nosemaphore at ffffffff8106ed63
#6 [ffff884187a039d0] __do_page_fault at ffffffff8106f2e8
#7 [ffff884187a03a40] do_page_fault at ffffffff8106f5f7
#8 [ffff884187a03a80] page_fault at ffffffff816bd608
[exception RIP: memset_erms+9]
RIP: ffffffff81332c69 RSP: ffff884187a03b30 RFLAGS: 00010246
RAX: ffff884187a04000 RBX: 000000000000004c RCX: 000000000000004c
RDX: 000000000000004c RSI: 0000000000000000 RDI: ffff885020478000
RBP: ffff884187a03b48 R8: 0000000000000000 R9: ffff885020478000
R10: 00000000017fffff R11: ffffffff819849c0 R12: 0000000000000000
R13: ffff884187a03d38 R14: 0000000000000140 R15: 00007f06e5e7c828
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
#9 [ffff884187a03b30] copy_user_handle_tail at ffffffff81333048
#10 [ffff884187a03b50] copy_page_from_iter_iovec at ffffffff81338401
#11 [ffff884187a03bc0] copy_page_from_iter at ffffffff813387c1
#12 [ffff884187a03bd0] rds_message_copy_from_user at ffffffffa04e62ae [rds]
#13 [ffff884187a03c20] rds_sendmsg at ffffffffa04ea268 [rds]
#14 [ffff884187a03cf0] sock_sendmsg at ffffffff815c094d
#15 [ffff884187a03d10] ___sys_sendmsg at ffffffff815c32ca
#16 [ffff884187a03eb0] __sys_sendmsg at ffffffff815c34f9
#17 [ffff884187a03f40] sys_sendmsg at ffffffff815c3559
#18 [ffff884187a03f50] system_call_fastpath at ffffffff816b9894
RIP: 00007f06e3bfae00 RSP: 00007fffa61d73f8 RFLAGS: 00000246
RAX: ffffffffffffffda RBX: 000000000ccb4e20 RCX: 00007f06e3bfae00
RDX: 0000000000000000 RSI: 00007fffa61d7400 RDI: 000000000000000b
RBP: 00007fffa61d7490 R8: 000000000ccb5410 R9: 0000000000000004
R10: 0000000000000000 R11: 0000000000000246 R12: 000000000cc98240
R13: 000000000ccb4e20 R14: 000000000cc98240 R15: 0000000000000001
ORIG_RAX: 000000000000002e CS: 0033 SS: 002b

> kmem -i
PAGES TOTAL PERCENTAGE
TOTAL MEM 131948662 503.3 GB ----
FREE 1618876 6.2 GB 1% of TOTAL MEM
USED 130329786 497.2 GB 98% of TOTAL MEM
SHARED 6239983 23.8 GB 4% of TOTAL MEM
BUFFERS 324526 1.2 GB 0% of TOTAL MEM
CACHED 13232100 50.5 GB 10% of TOTAL MEM
SLAB 1968657 7.5 GB 1% of TOTAL MEM

TOTAL SWAP 6291455 24 GB ----
SWAP USED 474120 1.8 GB 7% of TOTAL SWAP
SWAP FREE 5817335 22.2 GB 92% of TOTAL SWAP

COMMIT LIMIT 26185786 99.9 GB ----
COMMITTED 21063418 80.4 GB 80% of TOTAL LIMIT

> log

[20481093.748229] OKSK-00025: Cluster membership node count: 4, Local Node
Number: 3.
[20481093.748254] ADVMK-0013: Cluster reconfiguration started.
[20481095.300644] ADVMK-0014: Cluster reconfiguration completed.
[20481095.300649] ADVMK-0014: Cluster reconfiguration completed.
[20481095.301806] OKSK-00009: Cluster Membership Change complete.
[20543347.924475] RDS/IB: connection <192.168.32.106,192.168.32.89,0> dropped due to 'DISCONNECTED event'
[20543347.965472] RDS/IB: Active conn ffff8810e55f9d10 i_cm_id ffff880a645e5c00, frag 16KB, connected <192.168.32.106,192.168.32.89,0> version 4.1
[20550654.929211] RDS/IB: connection <192.168.32.105,192.168.32.90,0> dropped due to 'DISCONNECTED event'
[20550654.958371] RDS/IB: Passive conn ffff8801852a8000 i_cm_id ffff88064d906400, frag 16KB, connected <192.168.32.105,192.168.32.90,0> version 4.1
[20550688.698375] megaraid_sas 0000:23:00.0: Application firmware crash dump mode set success
[20550688.700012] megaraid_sas 0000:23:00.0: Application firmware crash dump mode set success
[20550689.410307] megaraid_sas 0000:23:00.0: Application firmware crash dump mode set success
[20583299.672370] BUG: unable to handle kernel paging request at ffff885020478000
[20583299.680485] IP: [<ffffffff81332c69>] memset_erms+0x9/0x10
[20583299.686841] PGD 208c067 PUD 5027df2063 PMD 428c532063 PTE 8000005020478061
[20583299.694873] Oops: 0003 [#1] SMP
[20583299.698788] Modules linked in: rpcsec_gss_krb5 nfsv4 oracleacfs(PO) oracleadvm(PO) oracleoks(PO) ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter ip_tables xt_conntrack nf_nat nf_conntrack br_netfilter bridge stp llc overlay nfsd auth_rpcgss autofs4 nfsv3 nfs_acl nfs fscache lockd grace sunrpc ipmi_poweroff ipmi_devintf bonding rds_rdma rds ib_sdp ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad dm_multipath bnx2i cnic uio cxgb4i libcxgbi cxgb4 ib_iser rdma_cm ib_cm iw_cm iscsi_tcp libiscsi_tcp libiscsi
scsi_transport_iscsi fuse iTCO_wdt iTCO_vendor_support sb_edac edac_core i2c_i801 lpc_ich mfd_core ioatdma sg ipmi_ssif i2c_core ipmi_si ipmi_msghandler ixgbe mlx4_ib ib_sa dca ib_mad ib_core ptp ib_addr pps_core vxlan ipv6 udp_tunnel
[20583299.779194] ip6_udp_tunnel mdio mlx4_core wmi ext4 jbd2 mbcache2 sd_mod ahci libahci megaraid_sas dm_mirror dm_region_hash dm_log dm_mod [last unloaded: oracleoks]
[20583299.794378] CPU: 67 PID: 342933 Comm: oracle Tainted: P O 4.1.12-94.7.8.el6uek.x86_64 #2
[20583299.804993] Hardware name: Oracle Corporation ORACLE SERVER X5-2/ASM,MOTHERBOARD,1U, BIOS 30130200 09/21/2017
[20583299.816389] task: ffff88418dc40000 ti: ffff884187a00000 task.ti: ffff884187a00000
[20583299.825426] RIP: 0010:[<ffffffff81332c69>] [<ffffffff81332c69>] memset_erms+0x9/0x10
[20583299.834869] RSP: 0018:ffff884187a03b30 EFLAGS: 00010246
[20583299.841295] RAX: ffff884187a04000 RBX: 000000000000004c RCX: 000000000000004c
[20583299.849940] RDX: 000000000000004c RSI: 0000000000000000 RDI: ffff885020478000
[20583299.858582] RBP: ffff884187a03b48 R08: 0000000000000000 R09: ffff885020478000
[20583299.867269] R10: 00000000017fffff R11: ffffffff819849c0 R12: 0000000000000000
[20583299.875913] R13: ffff884187a03d38 R14: 0000000000000140 R15: 00007f06e5e7c828
[20583299.884558] FS: 00007f06e5eaf700(0000) GS:ffff887f7e1c0000(0000) knlGS:0000000000000000
[20583299.894277] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[20583299.901186] CR2: ffff885020478000 CR3: 000000411dc32000 CR4: 0000000000160670
[20583299.909830] Stack:
[20583299.912555] ffffffff81333048 0000000000010286 ffff884187a03ec8 ffff884187a03bb8
[20583299.921557] ffffffff81338401 ffff884a45feb3d0 ffff885020478000 000000000000004c
[20583299.930535] ffff885020478000 ffff884187a03bc8 0400000000000286 0000000000000286
[20583299.939552] Call Trace:
[20583299.942817] [<ffffffff81333048>] ? copy_user_handle_tail+0x88/0xa0
[20583299.950310] [<ffffffff81338401>] copy_page_from_iter_iovec+0x1a1/0x2b0
[20583299.958192] [<ffffffff813387c1>] copy_page_from_iter+0x11/0x60
[20583299.965313] [<ffffffffa04e62ae>] rds_message_copy_from_user+0xce/0x130 [rds]
[20583299.973965] [<ffffffffa04ea268>] rds_sendmsg+0x468/0xa70 [rds]
[20583299.981080] [<ffffffff8120bef0>] ? rw_copy_check_uvector+0xa0/0x130
[20583299.988678] [<ffffffff815c094d>] sock_sendmsg+0x4d/0x60
[20583299.995105] [<ffffffff815c32ca>] ___sys_sendmsg+0x30a/0x330
[20583300.001931] [<ffffffffa0538ac6>] ? rds_ib_get_mr+0xd6/0x1a0 [rds_rdma]
[20583300.009822] [<ffffffffa04ed32a>] ? __rds_rdma_map+0x16a/0x340 [rds]
[20583300.017422] [<ffffffff815c34f9>] __sys_sendmsg+0x49/0x90
[20583300.023946] [<ffffffff81025ad1>] ? syscall_trace_leave+0xf1/0x160
[20583300.031348] [<ffffffff815c3559>] SyS_sendmsg+0x19/0x20
[20583300.037680] [<ffffffff816b9894>] system_call_fastpath+0x12/0xce
[20583300.044877] Code: 48 c1 e9 03 40 0f b6 f6 48 b8 01 01 01 01 01 01 01 01 48 0f af c6 f3 48 ab 89 d1 f3 aa 4c 89 c8 c3 90 49 89 f9 40 88 f0 48 89 d1 <f3> aa 4c 89 c8 c3 90 49 89 fa 40 0f b6 ce 48 b8 01 01 01 01 01
[20583300.067674] RIP [<ffffffff81332c69>] memset_erms+0x9/0x10
[20583300.074317] RSP <ffff884187a03b30>
[20583300.078703] CR2: ffff885020478000

Similar logs from the console logs:


-- xxxxxxxx.xxx.xxxxxxxx.xxx.xx.SPConsoleHistory.log (Console logs)
xxxxxxxx.xxx.xxxxxxxx.xxx.xx login: [20583299.672370] BUG: unable to handle kernel paging request at ffff885020478000
[20583299.680485] IP: [<ffffffff81332c69>] memset_erms+0x9/0x10
[20583299.686841] PGD 208c067 PUD 5027df2063 PMD 428c532063 PTE 8000005020478061
[20583299.694873] Oops: 0003 [#1] SMP
[20583299.698788] Modules linked in: rpcsec_gss_krb5 nfsv4 oracleacfs(PO) oracleadvm(PO) oracleoks(PO) ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter ip_tables xt_conntrack nf_nat nf_conntrack br_netfilter bridge stp llc overlay nfsd auth_rpcgss autofs4 nfsv3 nfs_acl nfs fscache lockd
grace sunrpc ipmi_poweroff ipmi_devintf bonding rds_rdma rds ib_sdp ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad dm_multipath bnx2i cnic uio cxgb4i libcx gbi cxgb4 ib_iser rdma_cm ib_cm iw_cm iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse iTCO_wdt iTCO_vendor_support sb_edac edac_core i2c_i801 lpc_ich mfd_core ioatdma sg ipmi_ssif i2c_core ipmi_si ipmi_msghandler ixgbe mlx4_ib ib_sa dca ib_mad ib_core ptp ib_addr pps_core vxlan ipv6 udp_tunnel ip6_udp_tunnel mdio mlx4_core wmi ext4 jbd2 mbcache2 sd_mod ahci libahci megaraid_sas dm_mirror dm_region_hash dm_log dm_mod [last unloaded: oracleoks]
[20583299.794378] CPU: 67 PID: 342933 Comm: oracle Tainted: P O 4.1.12-94.7.8.el6uek.x86_64 #2
[20583299.804993] Hardware name: Oracle Corporation ORACLE SERVER X5-2/ASM,MOTHERBOARD,1U, BIOS 30130200 09/21/2017
[20583299.816389] task: ffff88418dc40000 ti: ffff884187a00000 task.ti: ffff884187a00000
[20583299.825426] RIP: 0010:[<ffffffff81332c69>] [<ffffffff81332c69>] memset_erms+0x9/0x10
[20583299.834869] RSP: 0018:ffff884187a03b30 EFLAGS: 00010246 [20583299.841295] RAX: ffff884187a04000 RBX: 000000000000004c RCX: 000000000000004c
[20583299.849940] RDX: 000000000000004c RSI: 0000000000000000 RDI: ffff885020478000
[20583299.858582] RBP: ffff884187a03b48 R08: 0000000000000000 R09: ffff885020478000
[20583299.867269] R10: 00000000017fffff R11: ffffffff819849c0 R12: 0000000000000000
[20583299.875913] R13: ffff884187a03d38 R14: 0000000000000140 R15: 00007f06e5e7c828
[20583299.884558] FS: 00007f06e5eaf700(0000) GS:ffff887f7e1c0000(0000) knlGS:0000000000000000
[20583299.894277] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[20583299.901186] CR2: ffff885020478000 CR3: 000000411dc32000 CR4: 0000000000160670
[20583299.909830] Stack:
[20583299.912555] ffffffff81333048 0000000000010286 ffff884187a03ec8 ffff884187a03bb8
[20583299.921557] ffffffff81338401 ffff884a45feb3d0 ffff885020478000 000000000000004c
[20583299.930535] ffff885020478000 ffff884187a03bc8 0400000000000286 0000000000000286
[20583299.939552] Call Trace:
[20583299.942817] [<ffffffff81333048>] ? copy_user_handle_tail+0x88/0xa0
[20583299.950310] [<ffffffff81338401>] copy_page_from_iter_iovec+0x1a1/0x2b0
[20583299.958192] [<ffffffff813387c1>] copy_page_from_iter+0x11/0x60
[20583299.965313] [<ffffffffa04e62ae>] rds_message_copy_from_user+0xce/0x130 [rds]
[20583299.973965] [<ffffffffa04ea268>] rds_sendmsg+0x468/0xa70 [rds]
[20583299.981080] [<ffffffff8120bef0>] ? rw_copy_check_uvector+0xa0/0x130
[20583299.988678] [<ffffffff815c094d>] sock_sendmsg+0x4d/0x60
[20583299.995105] [<ffffffff815c32ca>] ___sys_sendmsg+0x30a/0x330
[20583300.001931] [<ffffffffa0538ac6>] ? rds_ib_get_mr+0xd6/0x1a0 [rds_rdma]
[20583300.009822] [<ffffffffa04ed32a>] ? __rds_rdma_map+0x16a/0x340 [rds]
[20583300.017422] [<ffffffff815c34f9>] __sys_sendmsg+0x49/0x90
[20583300.023946] [<ffffffff81025ad1>] ? syscall_trace_leave+0xf1/0x160
[20583300.031348] [<ffffffff815c3559>] SyS_sendmsg+0x19/0x20
[20583300.037680] [<ffffffff816b9894>] system_call_fastpath+0x12/0xce
[20583300.044877] Code: 48 c1 e9 03 40 0f b6 f6 48 b8 01 01 01 01 01 01 01 01 48 0f af c6 f3 48 ab 89 d1 f3 aa 4c 89 c8 c3 90 49 89 f9 40 88 f0 48 89 d1
<f3> aa 4c 89 c8 c3 90 49 89 fa 40 0f b6 ce 48 b8 01 01 01 01 01
[20583300.067674] RIP [<ffffffff81332c69>] memset_erms+0x9/0x10
[20583300.074317] RSP <ffff884187a03b30>
[20583300.078703] CR2: ffff885020478000
[ 0.000000] Initializing cgroup subsys cpuset
[ 0.000000] Initializing cgroup subsys cpu

 

-- messages: 
Mar 6 11:23:43 xxxxxxxx nscd: 181458 monitoring directory '/etc' (2)
Mar 6 11:23:43 xxxxxxxx nscd: 181458 monitoring file '/etc/services' (6)
Mar 6 11:23:43 xxxxxxxx nscd: 181458 monitoring directory '/etc' (2)
Mar 6 12:14:03 xxxxxxxx kernel: imklog 5.8.10, log source = /proc/kmsg started.
Mar 6 12:14:03 xxxxxxxx rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="13366" x-info="http://www.rsyslog.com"] start
Mar 6 12:14:03 xxxxxxxx kernel: [ 0.000000] Initializing cgroup subsys cpuset
Mar 6 12:14:03 xxxxxxxx kernel: [ 0.000000] Initializing cgroup subsys cpu
Mar 6 12:14:03 xxxxxxxx kernel: [ 0.000000] Initializing cgroup subsys cpuacct
Mar 6 12:14:03 xxxxxxxx kernel: [ 0.000000] Linux version 4.1.12-94.7.8.el6uek.x86_64 (mockbuild@x86-ol6-builder-04) (gcc version 4.4.7
20120313 (Red Hat 4.4.7-11) (GCC) ) #2 SMP Thu Ja Mar 6 12:14:03 xxxxxxxx kernel: [ 0.000000] Command line:
BOOT_IMAGE=/vmlinuz-4.1.12-94.7.8.el6uek.x86_64 root=/dev/mapper/VGExaDb-LVDbSys1 ro root=LABEL=DBSYS bootarea=dbsys bootfrom=B

 

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


In this Document
Symptoms
Cause
Solution
References


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.