Crash of RAC Node with "BUG: unable to handle kernel NULL pointer dereference at 0000000000000008" (free_pidmap.isra.0+0x21/0x30) (Doc ID 2503241.1)

Last updated on APRIL 24, 2020

Applies to:

Symptoms

One of the nodes which is part of a RAC crashes/panics frequently. This crash/behavior was observed on both Oracle Linux 6 and Oracle Linux 7 node RACs with RHCK and kmod-oracleasm module.

1. Oracle Linux 7 node (RHCK)
   
   PANIC: "BUG: unable to handle kernel NULL pointer dereference at 0000000000000008"

   crash7lates> log [293151.016335] BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
   [293151.016604] IP: [<ffffffff8dcb75e1>] free_pidmap.isra.0+0x21/0x30
   [293151.016803] PGD 800000e768ba0067 PUD fb8f1f1067 PMD 0
   [293151.016966] Oops: 0002 [#1] SMP
   [293151.017069] Modules linked in: macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag oracleacfs(POE) oracleadvm(POE) oracleoks(POE) mptctl mptbase bonding oracleasm(O) sunrpc ext4 mbcache jbd2 iTCO_wdt gpio_ich iTCO_vendor_support intel_powerclamp coretemp kvm_intel kvm irqbypass pcspkr hpwdt hpilo lpc_ich ses enclosure sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter shpchp binfmt_misc ip_tables xfs libcrc32c dm_service_time sd_mod sr_mod cdrom amdkfd amd_iommu_v2 radeon lpfc ata_generic pata_acpi dm_multipath nvmet_fc(T) i2c_algo_bit nvmet crc_t10dif drm_kms_helper crct10dif_generic hpsa syscopyarea sysfillrect nvme_fc(T) sysimgblt ata_piix fb_sys_fops nvme_fabrics e1000e ttm nvme_core drm scsi_transport_fc ptp scsi_tgt libata crc32c_intel be2net i2c_core netxen_nic
   [293151.031513] serio_raw bnx2 scsi_transport_sas pps_core crct10dif_common dm_mirror dm_region_hash dm_log dm_mod
   [293151.040737] CPU: 18 PID: 90553 Comm: disk_usage_repo Kdump: loaded Tainted: P OE ------------ T 3.10.0-862.9.1.el7.x86_64 #1
   [293151.050513] Hardware name: HP ProLiant DL980 G7, BIOS P66 08/16/2015
   [293151.055680] task: ffff8d0715f95ee0 ti: ffff8d435b7ec000 task.ti: ffff8d435b7ec000
   [293151.061036] RIP: 0010:[<ffffffff8dcb75e1>] [<ffffffff8dcb75e1>] free_pidmap.isra.0+0x21/0x30
   [293151.066628] RSP: 0018:ffff8d435b7efd28 EFLAGS: 00010082
   [293151.072260] RAX: ffffffff8e8a44b0 RBX: ffff8d20ecfeb800 RCX: ffff8d20ecfeb830
   [293151.078092] RDX: 0000000000000000 RSI: ffffffff8e84fde0 RDI: 0000000000000058
   [293151.084032] RBP: ffff8d435b7efd28 R08: ffff8ce85b7d44d8 R09: ffff8d435b7efd48
   [293151.090033] R10: 0000000000000000 R11: ffff8d435b7efd50 R12: 0000000000000001
   [293151.096143] R13: 0000000000000046 R14: ffff8c6978bab180 R15: 0000000000000000
   [293151.102310] FS: 00007fbe91a6f740(0000) GS:ffff8cbe3f880000(0000) knlGS:0000000000000000
   [293151.108520] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
   [293151.114820] CR2: 0000000000000008 CR3: 000000e4a3a40000 CR4: 00000000000007e0
   [293151.121286] Call Trace:
   [293151.127781] [<ffffffff8dcb776b>] free_pid+0xdb/0x100
   [293151.134438] [<ffffffff8dcb77ea>] __change_pid+0x5a/0x60
   [293151.141164] [<ffffffff8dcb7d20>] detach_pid+0x10/0x20
   [293151.147910] [<ffffffff8dc96946>] release_task+0x246/0x490
   [293151.154734] [<ffffffff8dc974ca>] wait_consider_task+0x93a/0xb30
   [293151.161651] [<ffffffff8dc977c0>] do_wait+0x100/0x260
   [293151.168562] [<ffffffff8dc989d0>] SyS_wait4+0x80/0x110
   [293151.175525] [<ffffffff8dc96510>] ? task_stopped_code+0x60/0x60
   [293151.182589] [<ffffffff8e320795>] system_call_fastpath+0x1c/0x21
   [293151.189657] [<ffffffff8e3206e1>] ? system_call_after_swapgs+0xae/0x146
   [293151.196833] Code: 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 48 63 c7 55 81 e7 ff 7f 00 00 48 c1 e8 0f 48 c1 e0 04 48 89 e5 48 01 f0 48 8b 50 10 <f0> 0f b3 3a f0 ff 40 08 5d c3 0f 1f 44 00 00 66 66 66 66 90 81
   [293151.212558] RIP [<ffffffff8dcb75e1>] free_pidmap.isra.0+0x21/0x30
   [293151.220453] RSP <ffff8d435b7efd28>
   [293151.228386] CR2: 0000000000000008

   
   
2. Coredump analysis of Oracle Linux 6 node (RHCK)
   
   PANIC: "general protection fault: 0000 [#1] SMP "
   
   
   [695605.569370] general protection fault: 0000 [#1] SMP
   [695605.571892] Modules linked in: hangcheck_timer mptctl mptbase oracleasm sunrpc bonding ipv6 iTCO_wdt iTCO_vendor_support serio_raw pcspkr tg3 ptp pps_core ipmi_devintf ipmi_si ipmi_msghandler lpc_ich mfd_core hpwdt hpilo shpchp sg pcc_cpufreq acpi_cpufreq ext4 jbd2 mbcache2 dm_round_robin sd_mod hpsa scsi_transport_sas be2net vxlan udp_tunnel ip6_udp_tunnel lpfc scsi_transport_fc radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core dm_multipath dm_mirror dm_region_hash dm_log dm_mod
   [695605.594498] CPU: 5 PID: 75623 Comm: hpetfe Not tainted 4.1.12-124.16.4.el6uek.x86_64 #2
   [695605.598207] Hardware name: HP ProLiant BL680c G7, BIOS I25 02/23/2018
   [695605.601447] task: ffff883fabe0e200 ti: ffff882821f40000 task.ti: ffff882821f40000
   [695605.605097] RIP: 0010:[<ffffffff810a21bd>] [<ffffffff810a21bd>] free_pidmap.isra.0+0x1d/0x30
   [695605.609151] RSP: 0018:ffff882821f43cf8 EFLAGS: 00010007
   [695605.611700] RAX: 001ffffffffb8ed0 RBX: ffff88088eba2400 RCX: ffff88088eba2400
   [695605.615091] RDX: ffff88803ffb18c8 RSI: 001fffff81ad2850 RDI: 0000000000004590
   [695605.618549] RBP: ffff882821f43cf8 R08: 0000000000000000 R09: ffff882821f43d28
   [695605.622000] R10: 0000000000000000 R11: 000000000000001a R12: 0000000000000001
   [695605.626276] R13: ffffffff81b19980 R14: 0000000000000001 R15: ffff881fb0b2b800
   [695605.629728] FS: 00007fb59eec4700(0000) GS:ffff881fbf940000(0000) knlGS:0000000000000000
   [695605.633702] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
   [695605.636703] CR2: 00000000006df5ec CR3: 000000281650a000 CR4: 0000000000020670
   [695605.643902] Stack:
   [695605.648520] ffff882821f43d28 ffffffff810a23ab 0000000000000001 ffff8808b03bf380
   [695605.656112] ffff88283a232880 ffff88283a232a34 ffff882821f43d38 ffffffff810a2461
   [695605.663500] ffff882821f43d48 ffffffff810a2ae0 ffff882821f43db8 ffffffff81085b00
   [695605.670931] Call Trace:
   [695605.675706] [<ffffffff810a23ab>] free_pid+0x11b/0x160
   [695605.681780] [<ffffffff810a2461>] __change_pid+0x71/0x80
   [695605.687998] [<ffffffff810a2ae0>] detach_pid+0x10/0x20
   [695605.694061] [<ffffffff81085b00>] release_task+0x220/0x470
   [695605.700440] [<ffffffff8108623a>] wait_consider_task+0x4ea/0xcc0
   [695605.706842] [<ffffffff81086b10>] do_wait+0x100/0x270
   [695605.712692] [<ffffffff81087cd3>] do_wait4+0x63/0xe0
   [695605.718421] [<ffffffff810856a0>] ? task_stopped_code+0x60/0x60
   [695605.724647] [<ffffffff81087d6d>] SyS_wait4+0x1d/0x20
   [695605.730651] [<ffffffff816edc5c>] system_call_fastpath+0x18/0xd6
   [695605.736959] Code: 5d c3 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 66 66 66 66 90 48 63 c7 81 e7 ff 7f 00 00 48 c1 e8 0f 48 c1 e0 04 48 01 c6 <48> 8b 46 10 f0 48 0f b3 38 f0 ff 46 08 5d c3 0f 1f 40 00 55 48
   [695605.756634] RIP [<ffffffff810a21bd>] free_pidmap.isra.0+0x1d/0x30
   [695605.763273] RSP <ffff882821f43cf8>
   

Cause

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.