Oracle VM Server Crashed with "i40e_notify_client_of_l2_param_changes+0xbd/0x1f0 [i40e]"
(Doc ID 2239604.1)
Last updated on JUNE 09, 2021
Applies to:
Oracle VM - Version 3.4.1 to 3.4.2 [Release OVM34]Oracle Cloud Infrastructure - Version N/A and later
Linux x86-64
Symptoms
Oracle VM servers are connected to an ISCSI storage array with Intel X710 Fortville network adapters.
- 85:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)
85:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)
When you restart one of switches in the Oracle VM server ISCSI network for maintenance job, the Oracle VM server crashes.
When you examine the vmcore generated from the crash, you see below call trace:
- crash6latest> bt
PID: 715 TASK: ffff8800059ac600 CPU: 18 COMMAND: "kworker/u40:7"
#0 [ffff8800059c3880] panic at ffffffff816c67f9
#1 [ffff8800059c3900] oops_end at ffffffff8101a79c
#2 [ffff8800059c3930] no_context at ffffffff8106d7a1
#3 [ffff8800059c3980] __bad_area_nosemaphore at ffffffff8106d99d
#4 [ffff8800059c39d0] bad_area_nosemaphore at ffffffff8106dab3
#5 [ffff8800059c39e0] __do_page_fault at ffffffff8106e028
#6 [ffff8800059c3a50] do_page_fault at ffffffff8106e337
#7 [ffff8800059c3a90] page_fault at ffffffff816cd758
[exception RIP: i40e_notify_client_of_l2_param_changes+189]
RIP: ffffffffa02c9aad RSP: ffff8800059c3b48 RFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff8803404d4000 RCX: 0000000000000003
RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff880005ec0008
RBP: ffff8800059c3b98 R8: 0000000000000000 R9: 0000000000000000
R10: 0000000000000001 R11: ffff8800059c3b6a R12: ffff8800059c3b48
R13: ffff880005ec0000 R14: 0000000000000001 R15: ffff880005ec0688
ORIG_RAX: ffffffffffffffff CS: e030 SS: e02b
#8 [ffff8800059c3ba0] i40e_dcb_reconfigure at ffffffffa02a803a [i40e]
#9 [ffff8800059c3bd0] i40e_handle_lldp_event at ffffffffa02afcc0 [i40e]
#10 [ffff8800059c3cf0] i40e_clean_adminq_subtask at ffffffffa02b028d [i40e]
#11 [ffff8800059c3d90] i40e_service_task at ffffffffa02b08dc [i40e]
#12 [ffff8800059c3de0] process_one_work at ffffffff8109f07e
#13 [ffff8800059c3e40] worker_thread at ffffffff8109f500
#14 [ffff8800059c3ec0] kthread at ffffffff810a46de
#15 [ffff8800059c3f50] ret_from_fork at ffffffff816cbb62
crash6latest> - [ 692.916083] i40e 0000:01:00.1 em2: NIC Link is Down
[ 693.168489] BUG: unable to handle kernel NULL pointer dereference at 0000000000000230
[ 693.168742] IP: [<ffffffffa02c9aad>] i40e_notify_client_of_l2_param_changes+0xbd/0x1f0 [i40e]
[ 693.169304] PGD 33ab49067 PUD 34961f067 PMD 0
[ 693.169550] Oops: 0000 [#1] SMP
[ 693.169788] Modules linked in: xen_netback xen_blkback xen_gntalloc xen_gntdev xen_evtchn xenfs xen_privcmd ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue configfs bnx2fc fcoe libfcoe libfc scsi_transport_fc bridge 8021q mrp garp stp llc bonding ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp dm_round_robin dm_multipath iTCO_wdt iTCO_vendor_support dcdbas pcspkr sb_edac edac_core lpc_ich mfd_core shpchp ipmi_devintf ipmi_si ipmi_msghandler sg ext4 jbd2 mbcache sr_mod cdrom sd_mod mxm_wmi ahci libahci igb dca i2c_algo_bit i2c_core i40e ptp pps_core vxlan udp_tunnel ip6_udp_tunnel megaraid_sas wmi crc32c_intel be2iscsi bnx2i cnic uio cxgb4i cxgb4 cxgb3i libcxgbi ipv6 cxgb3 mdio libiscsi_tcp qla4xxx iscsi_boot_sysfs libiscsi scsi_transport_iscsi dm_mirror
[ 693.172555] dm_region_hash dm_log dm_mod
[ 693.172800] CPU: 18 PID: 715 Comm: kworker/u40:7 Not tainted 4.1.12-61.1.16.el6uek.x86_64 #2
[ 693.173261] Hardware name: Dell Inc. PowerEdge R730/072T6D, BIOS 2.1.5 04/11/2016
[ 693.173724] Workqueue: i40e i40e_service_task [i40e]
[ 693.173968] task: ffff8800059ac600 ti: ffff8800059c0000 task.ti: ffff8800059c0000
[ 693.174434] RIP: e030:[<ffffffffa02c9aad>] [<ffffffffa02c9aad>] i40e_notify_client_of_l2_param_changes+0xbd/0x1f0 [i40e]
[ 693.174910] RSP: e02b:ffff8800059c3b48 EFLAGS: 00010246
[ 693.175147] RAX: 0000000000000000 RBX: ffff8803404d4000 RCX: 0000000000000003
[ 693.175388] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff880005ec0008
[ 693.175629] RBP: ffff8800059c3b98 R08: 0000000000000000 R09: 0000000000000000
[ 693.175870] R10: 0000000000000001 R11: ffff8800059c3b6a R12: ffff8800059c3b48
[ 693.176110] R13: ffff880005ec0000 R14: 0000000000000001 R15: ffff880005ec0688
[ 693.176355] FS: 0000000000000000(0000) GS:ffff880353080000(0000) knlGS:ffff880353080000
[ 693.176816] CS: e033 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 693.177054] CR2: 0000000000000230 CR3: 00000003481f7000 CR4: 0000000000042660
[ 693.177294] Stack:
[ 693.177522] 0000000300000003 0000000300000003 0000000300000003 0000000300000003
[ 693.177998] 0000000000000000 000000009112b991 ffff880005ec0000 0000000000000001
[ 693.178476] 0000000000000001 0000000000000001 ffff8800059c3bc8 ffffffffa02a803a
[ 693.178954] Call Trace:
[ 693.179190] [<ffffffffa02a803a>] i40e_dcb_reconfigure+0xda/0x1f0 [i40e]
[ 693.179431] [<ffffffffa02afcc0>] i40e_handle_lldp_event+0x2a0/0x440 [i40e]
[ 693.179677] [<ffffffffa02b028d>] i40e_clean_adminq_subtask+0x42d/0x580 [i40e]
[ 693.180149] [<ffffffffa02b08dc>] i40e_service_task+0x16c/0x240 [i40e]
[ 693.180396] [<ffffffff8109f07e>] process_one_work+0x14e/0x4b0
[ 693.180638] [<ffffffff8100c262>] ? check_events+0x12/0x20
[ 693.180875] [<ffffffff8109f500>] worker_thread+0x120/0x480
[ 693.181114] [<ffffffff816c6ea9>] ? __schedule+0x309/0x880
[ 693.181351] [<ffffffff8109f3e0>] ? process_one_work+0x4b0/0x4b0
[ 693.181589] [<ffffffff8109f3e0>] ? process_one_work+0x4b0/0x4b0
[ 693.181831] [<ffffffff810a46de>] kthread+0xce/0xf0
[ 693.182070] [<ffffffff810a4610>] ? kthread_freezable_should_stop+0x70/0x70
[ 693.182310] [<ffffffff816cbb62>] ret_from_fork+0x42/0x70
[ 693.182546] [<ffffffff810a4610>] ? kthread_freezable_should_stop+0x70/0x70
[ 693.182786] Code: 63 ca 0f b7 8c 4b 04 06 00 00 40 88 30 66 83 f9 ff 66 89 48 fe 0f 84 1b 01 00 00 48 83 c0 04 48 83 c7 01 4c 39 d8 75 b6 48 8b 03 <8b> 80 30 02 00 00 66 89 45 d0 48 c7 c7 80 13 2e a0 e8 8d f8 3f
[ 693.183905] RIP [<ffffffffa02c9aad>] i40e_notify_client_of_l2_param_changes+0xbd/0x1f0 [i40e]
[ 693.184369] RSP <ffff8800059c3b48>
[ 693.184598] CR2: 0000000000000230
[ 693.185183] ---[ end trace c8995ddd06f149d9 ]---
[ 693.191564] Kernel panic - not syncing: Fatal exception
[ 693.191876] Kernel Offset: disabled
crash6latest>
Changes
Restart the switch in OVM server's ISCSI network.
Cause
To view full details, sign in with your My Oracle Support account. |
|
Don't have a My Oracle Support account? Click to get started! |
In this Document
| Symptoms |
| Changes |
| Cause |
| Solution |
| References |