My Oracle Support Banner

OL6 - Server Unexpected crash with error "list_add double add" (Doc ID 2586682.1)

Last updated on SEPTEMBER 17, 2019

Applies to:

Linux OS - Version Oracle Linux 6.7 with Unbreakable Enterprise Kernel [4.1.12] and later
Linux x86-64

Symptoms

OL6 server running uek4 kernel crashed with following call trace.

<>

KERNEL: vmlinux
DUMPFILE: All_files/file1.tar [PARTIAL DUMP]
CPUS: 24
DATE: Thu Jul 25 01:32:55 2019
UPTIME: 17 days, 15:23:10
LOAD AVERAGE: 0.26, 0.30, 0.36
TASKS: 1419
NODENAME: localhost
RELEASE: 4.1.12-124.27.2.el6uek.x86_64
VERSION: #2 SMP Wed May 22 10:02:58 PDT 2019
MACHINE: x86_64 (3399 Mhz)
MEMORY: 127.9 GB
PANIC: "BUG: unable to handle kernel NULL pointer dereference at0000000000000008"
PID: 34927
COMMAND: "kworker/1:2"
TASK: ffff8815be289c00 [THREAD_INFO: ffff881f5fcbc000]
CPU: 1
STATE: TASK_RUNNING (PANIC)

> log
[ 6892.748878] perf interrupt took too long (2510 > 2500), loweringkernel.perf_event_max_sample_rate to 50000
[13736.398017] rport-4:0-0: blocked FC remote port time out: removing targetand saving binding
[23226.676281] perf interrupt took too long (5001 > 5000), loweringkernel.perf_event_max_sample_rate to 25000
[23711.245094] ------------[ cut here ]------------
[23711.245109] WARNING: CPU: 1 PID: 900 at lib/list_debug.c:36__list_add+0xa0/0xb0()
[23711.245113] list_add double add: new=ffff882032598f40,prev=ffff882032598f40, next=ffff882032583ea0.
[23711.245116] Modules linked in: oracleasm autofs4 bonding ipv6 dm_round_robin dm_multipath uinput iTCO_wdt iTCO_vendor_support dcdbas pcspkr ch osst st sb_edac edac_core shpchp igb i2c_algo_bit lpc_ich mfd_core ixgbedca ptp pps_core vxlan udp_tunnel ip6_udp_tunnel mdio sg ipmi_ssif i2c_core ipmi_si ipmi_msghandler ext4 jbd2 mbcache2 sr_mod cdrom sd_mod ahci libahci qla2xxx scsi_transport_fc megaraid_sas mxm_wmi wmi dm_mirror dm_region_hash dm_log dm_mod
[23711.245183] CPU: 1 PID: 900 Comm: qla2xxx_4_dpc Not tainted 4.1.12-124.27.2.el6uek.x86_64 #2
[23711.245187] Hardware name: Dell Inc. PowerEdge R730/072T6D, BIOS 2.4.3 01/17/2017
[23711.245190] 0000000000000000 ffff882030e4bc78 ffffffff816eba56 ffff882030e4bcc8
[23711.245196] ffffffff81a07321 ffff882030e4bcb8 ffffffff8108708a ffff882030c00000
[23711.245201] ffff882032598e00 ffff8820325837f8 0000000000000292 0000000000000102
[23711.245207] Call Trace:
[23711.245218] [<ffffffff816eba56>] dump_stack+0x63/0x81
[23711.245229] [<ffffffff8108708a>] warn_slowpath_common+0x8a/0xc0
[23711.245236] [<ffffffff81087106>] warn_slowpath_fmt+0x46/0x50
[23711.245244] [<ffffffff816ee43a>] ? __schedule+0x24a/0x810
[23711.245249] [<ffffffff81346930>] __list_add+0xa0/0xb0
[23711.245268] [<ffffffffc009cd66>] qla24xx_async_gnl+0xc6/0x420 [qla2xxx]
[23711.245276] [<ffffffff810cce78>] ? __wake_up+0x48/0x60
[23711.245282] [<ffffffff816ee43a>] ? __schedule+0x24a/0x810
[23711.245294] [<ffffffffc0094c5c>] qla2x00_do_work+0xfc/0x760 [qla2xxx]
[23711.245299] [<ffffffff816ee43a>] ? __schedule+0x24a/0x810
[23711.245305] [<ffffffff816ee42e>] ? __schedule+0x23e/0x810
[23711.245310] [<ffffffff816ee43a>] ? __schedule+0x24a/0x810
[23711.245315] [<ffffffff816ee42e>] ? __schedule+0x23e/0x810
[23711.245320] [<ffffffff816ee43a>] ? __schedule+0x24a/0x810
[23711.245326] [<ffffffff816ee46f>] ? __schedule+0x27f/0x810
[23711.245337] [<ffffffffc009567d>] qla2x00_do_dpc+0x14d/0x930 [qla2xxx]
[23711.245349] [<ffffffffc0095530>] ? qla2x00_relogin+0x230/0x230 [qla2xxx]
[23711.245358] [<ffffffff810a796b>] kthread+0xcb/0xf0
[23711.245363] [<ffffffff816ee43a>] ? __schedule+0x24a/0x810
[23711.245369] [<ffffffff816ee43a>] ? __schedule+0x24a/0x810
[23711.245374] [<ffffffff810a78a0>] ? kthread_create_on_node+0x180/0x180
[23711.245382] [<ffffffff816f45d1>] ret_from_fork+0x61/0x90
[23711.245387] [<ffffffff810a78a0>] ? kthread_create_on_node+0x180/0x180
[23711.245391] ---[ end trace c2b9e31ed4e7b3ed ]---

<>

Sep 10 04:16:54 localhost kernel: [5446259.010227] WARNING: CPU: 2 PID: 1497 at lib/list_debug.c:33 __list_add+0xa0/0xd0()
Sep 10 04:16:54 localhost kernel: [5446259.010230] list_add corruption. prev->next should be next (ffff8810377c6ea0), but was ffff8820300fed40. (prev=ffff8820300fed40).
Sep 10 04:16:54 localhost kernel: [5446259.010232] Modules linked in: oracleacfs(PO) oracleadvm(PO) oracleoks(PO) iptable_filter ip_tables dsa_filter(POE) oracleasm cpufreq_powersave bonding ipv6 uinput iTCO_wdt iTCO_vendor_support ipmi_devintf dcdbas sg ch osst st tg3 ptp pps_core intel_powerclamp coretemp kvm_intel kvm pcspkr sb_edac edac_core lpc_ich mfd_core ipmi_ssif i2c_core ipmi_si ipmi_msghandler shpchp ext4 jbd2 mbcache2 dm_round_robin sd_mod sr_mod cdrom qla2xxx scsi_transport_fc megaraid_sas ghash_clmulni_intel crc32_pclmul crc32c_intel aesni_intel ablk_helper cryptd lrw gf128mul glue_helper aes_x86_64 ahci libahci wmi dm_multipath dm_mirror dm_region_hash dm_log dm_mod
Sep 10 04:16:54 localhost kernel: [5446259.010292] CPU: 2 PID: 1497 Comm: qla2xxx_12_dpc Tainted: P W OE 4.1.12-124.28.3.el6uek.x86_64 #2
Sep 10 04:16:54 localhost kernel: [5446259.010294] Hardware name: Dell Inc. PowerEdge R730/0599V5, BIOS 1.3.6 06/03/2015
Sep 10 04:16:54 localhost kernel: [5446259.010296] 0000000000000000 ffff88202f43fc58 ffffffff816ebaee ffff88202f43fca8
Sep 10 04:16:54 localhost kernel: [5446259.010301] ffffffff81a083d1 ffff88202f43fc98 ffffffff8108708a ffff88102bea0000
Sep 10 04:16:54 localhost kernel: [5446259.010306] ffff8820300fe940 ffff8810377c6ea0 ffff8820300fed40 0000000000000102
Sep 10 04:16:54 localhost kernel: [5446259.010311] Call Trace:
Sep 10 04:16:54 localhost kernel: [5446259.010316] [<ffffffff816ebaee>] dump_stack+0x63/0x81
Sep 10 04:16:54 localhost kernel: [5446259.010322] [<ffffffff8108708a>] warn_slowpath_common+0x8a/0xc0
Sep 10 04:16:54 localhost kernel: [5446259.010328] [<ffffffff81087106>] warn_slowpath_fmt+0x46/0x50

Sep 10 04:16:54 localhost kernel: [5446259.010333] [<ffffffff81346a50>] __list_add+0xa0/0xd0
Sep 10 04:16:54 localhost kernel: [5446259.010345] [<ffffffffc0124d66>] qla24xx_async_gnl+0xc6/0x420 [qla2xxx]
Sep 10 04:16:54 localhost kernel: [5446259.010349] [<ffffffff810cce78>] ? __wake_up+0x48/0x60
Sep 10 04:16:54 localhost kernel: [5446259.010360] [<ffffffffc011cc5c>] qla2x00_do_work+0xfc/0x760 [qla2xxx]
Sep 10 04:16:54 localhost kernel: [5446259.010365] [<ffffffff816ee4ca>] ? __schedule+0x24a/0x810
Sep 10 04:16:54 localhost kernel: [5446259.010371] [<ffffffff816ee4be>] ? __schedule+0x23e/0x810
Sep 10 04:16:54 localhost kernel: [5446259.010376] [<ffffffff816ee4ca>] ? __schedule+0x24a/0x810
Sep 10 04:16:54 localhost kernel: [5446259.010381] [<ffffffff816ee4be>] ? __schedule+0x23e/0x810
Sep 10 04:16:54 localhost kernel: [5446259.010386] [<ffffffff816ee4ca>] ? __schedule+0x24a/0x810
Sep 10 04:16:54 localhost kernel: [5446259.010391] [<ffffffff816ee4ff>] ? __schedule+0x27f/0x810
Sep 10 04:16:54 localhost kernel: [5446259.010402] [<ffffffffc011d67d>] qla2x00_do_dpc+0x14d/0x930 [qla2xxx]
Sep 10 04:16:54 localhost kernel: [5446259.010412] [<ffffffffc011d530>] ? qla2x00_relogin+0x230/0x230 [qla2xxx]
Sep 10 04:16:54 localhost kernel: [5446259.010417] [<ffffffff810a796b>] kthread+0xcb/0xf0
Sep 10 04:16:54 localhost kernel: [5446259.010423] [<ffffffff816ee4ca>] ? __schedule+0x24a/0x810
Sep 10 04:16:54 localhost kernel: [5446259.010428] [<ffffffff816ee4ca>] ? __schedule+0x24a/0x810
Sep 10 04:16:54 localhost kernel: [5446259.010433] [<ffffffff810a78a0>] ? kthread_create_on_node+0x180/0x180
Sep 10 04:16:54 localhost kernel: [5446259.010438] [<ffffffff816f4661>] ret_from_fork+0x61/0x90
Sep 10 04:16:54 localhost kernel: [5446259.010442] [<ffffffff810a78a0>] ? kthread_create_on_node+0x180/0x180
Sep 10 04:16:54 localhost kernel: [5446259.010446] ---[ end trace 347be29085dfd3a5 ]---

Changes

 

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


In this Document
Symptoms
Changes
Cause
Solution
References


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.