My Oracle Support Banner

Oracle Linux: Running Sosreport Crashes Server "device-mapper: multipath: Failing path" "qla2xxx [0000:37:00.1]-801c:9: Abort command issued nexus=9:0:1 -- 0 2003" (Doc ID 2712836.1)

Last updated on SEPTEMBER 21, 2020

Applies to:

Linux OS - Version Oracle Linux 6.10 with Unbreakable Enterprise Kernel [4.1.12] and later
Linux x86-64

Symptoms

Server crashes whenever sosreport is triggered. A lot of the following error messages related to mpath and qlogic driver can be seen. 

Aug 19 05:40:35 HOSTNAME kernel: [496714.422674] qla2xxx [0000:37:00.0]-801c:8: Abort command issued nexus=8:3:39 -- 0 2003.
Aug 19 05:40:35 HOSTNAME kernel: [496714.518570] qla2xxx [0000:37:00.0]-801c:8: Abort command issued nexus=8:1:42 -- 0 2003.
Aug 19 05:40:48 HOSTNAME kernel: [496726.616311] device-mapper: multipath: Failing path 135:48.
Aug 19 05:40:48 HOSTNAME kernel: [496726.616358] device-mapper: multipath: Failing path 134:64.
Aug 19 05:40:48 HOSTNAME kernel: [496726.632678] device-mapper: multipath: Failing path 67:544.
Aug 19 05:40:48 HOSTNAME kernel: [496726.632729] device-mapper: multipath: Failing path 71:640.
Aug 19 05:40:48 HOSTNAME kernel: [496726.632774] device-mapper: multipath: Failing path 129:512.
Aug 19 05:40:48 HOSTNAME kernel: [496726.632822] device-mapper: multipath: Failing path 128:544.
Aug 19 05:41:01 HOSTNAME kernel: [496739.793684] device-mapper: multipath: Failing path 70:688.
Aug 19 05:41:15 HOSTNAME kernel: [496754.328355] qla2xxx [0000:37:00.0]-801c:8: Abort command issued nexus=8:1:0 -- 0 2003.
Aug 19 05:41:15 HOSTNAME kernel: [496754.424155] qla2xxx [0000:37:00.0]-801c:8: Abort command issued nexus=8:1:42 -- 0 2003.
Aug 19 05:41:32 HOSTNAME kernel: [496771.184390] hpilo 0000:01:00.2: Closing, but controller still active
Aug 19 05:41:55 HOSTNAME kernel: [496794.234349] qla2xxx [0000:37:00.0]-801c:8: Abort command issued nexus=8:0:0 -- 0 2003.
Aug 19 05:41:55 HOSTNAME kernel: [496794.329818] qla2xxx [0000:37:00.0]-801c:8: Abort command issued nexus=8:0:29 -- 0 2003.
Aug 19 05:42:24 HOSTNAME kernel: [496823.215830] qla2xxx [0000:37:00.1]-705e:9: Raw firmware dump ready for read on (9).
Aug 19 05:42:24 HOSTNAME kernel: [496823.218263] qla2xxx [0000:37:00.1]-705d:9: Firmware dump cleared on (9).
Aug 19 05:42:32 HOSTNAME kernel: [496830.814475] INFO: task parted:19784 blocked for more than 120 seconds.
Aug 19 05:42:32 HOSTNAME kernel: [496830.814478] Tainted: P O 4.1.12-124.41.1.bug31517449.el6uek.rc1.x86_64 #2
Aug 19 05:42:32 HOSTNAME kernel: [496830.814479] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug 19 05:42:32 HOSTNAME kernel: [496830.814480] parted D ffff885ebfdd8480 0 19784 19783 0x00000000
Aug 19 05:42:32 HOSTNAME kernel: [496830.814483] ffff886e2b52b898 0000000000000086 ffffffff816f17ea ffff887702b4d400
Aug 19 05:42:32 HOSTNAME kernel: [496830.814485] ffffffff816f17ea ffff886e2b52c000 ffff886e2b52ba00 7fffffffffffffff
Aug 19 05:42:32 HOSTNAME kernel: [496830.814487] ffff887702b4d400 ffff887702b4d400 ffff886e2b52b8b8 ffffffff816f1de7
Aug 19 05:42:32 HOSTNAME kernel: [496830.814488] Call Trace:
Aug 19 05:42:32 HOSTNAME kernel: [496830.814495] [<ffffffff816f17ea>] ? __schedule+0x24a/0x810
Aug 19 05:42:32 HOSTNAME kernel: [496830.814496] [<ffffffff816f17ea>] ? __schedule+0x24a/0x810
Aug 19 05:42:32 HOSTNAME kernel: [496830.814498] [<ffffffff816f1de7>] schedule+0x37/0x90
Aug 19 05:42:32 HOSTNAME kernel: [496830.814499] [<ffffffff816f17ea>] ? __schedule+0x24a/0x810
Aug 19 05:42:32 HOSTNAME kernel: [496830.814501] [<ffffffff816f562c>] schedule_timeout+0x24c/0x2d0
Aug 19 05:42:32 HOSTNAME kernel: [496830.814502] [<ffffffff816f181f>] ? __schedule+0x27f/0x810
Aug 19 05:42:32 HOSTNAME kernel: [496830.814503] [<ffffffff816f353c>] wait_for_completion+0x11c/0x180
Aug 19 05:42:32 HOSTNAME kernel: [496830.814507] [<ffffffff810b5f70>] ? wake_up_state+0x20/0x20
Aug 19 05:42:32 HOSTNAME kernel: [496830.814510] [<ffffffff810a189f>] flush_work+0xff/0x1f0
Aug 19 05:42:32 HOSTNAME kernel: [496830.814514] [<ffffffff8109e220>] ? destroy_worker+0xa0/0xa0
Aug 19 05:42:32 HOSTNAME kernel: [496830.814515] [<ffffffff810a1a98>] __cancel_work_timer+0x98/0x1b0
Aug 19 05:42:32 HOSTNAME kernel: [496830.814520] [<ffffffff81482288>] ? kobj_lookup+0x118/0x180
Aug 19 05:42:32 HOSTNAME kernel: [496830.814525] [<ffffffff8130e900>] ? disk_map_sector_rcu+0x80/0x80
Aug 19 05:42:32 HOSTNAME kernel: [496830.814526] [<ffffffff810a1be3>] cancel_delayed_work_sync+0x13/0x20
Aug 19 05:42:32 HOSTNAME kernel: [496830.814527] [<ffffffff813109b1>] disk_block_events+0x81/0x90
Aug 19 05:42:32 HOSTNAME kernel: [496830.814548] [<ffffffff81249eeb>] __blkdev_get+0x5b/0x490
Aug 19 05:42:32 HOSTNAME kernel: [496830.814550] [<ffffffff8124a35d>] blkdev_get+0x3d/0x380
Aug 19 05:42:32 HOSTNAME kernel: [496830.814551] [<ffffffff8124a75f>] blkdev_open+0x5f/0x90
Aug 19 05:42:32 HOSTNAME kernel: [496830.814557] [<ffffffff8120bcba>] do_dentry_open+0x1ea/0x340
Aug 19 05:42:32 HOSTNAME kernel: [496830.814558] [<ffffffff8124a700>] ? blkdev_get_by_dev+0x60/0x60
Aug 19 05:42:32 HOSTNAME kernel: [496830.814560] [<ffffffff8120d187>] vfs_open+0x57/0x80
Aug 19 05:42:32 HOSTNAME kernel: [496830.814563] [<ffffffff812192f8>] ? may_open+0x68/0x110
Aug 19 05:42:32 HOSTNAME kernel: [496830.814565] [<ffffffff8121c3ca>] do_last+0x21a/0x1210
Aug 19 05:42:32 HOSTNAME kernel: [496830.814566] [<ffffffff8121eda0>] path_openat+0x80/0x650
Aug 19 05:42:32 HOSTNAME kernel: [496830.814567] [<ffffffff816f181f>] ? __schedule+0x27f/0x810
Aug 19 05:42:32 HOSTNAME kernel: [496830.814568] [<ffffffff81220d5a>] do_filp_open+0x3a/0xb0
Aug 19 05:42:32 HOSTNAME kernel: [496830.814571] [<ffffffff8122ddc7>] ? __alloc_fd+0xa7/0x130
Aug 19 05:42:32 HOSTNAME kernel: [496830.814572] [<ffffffff8120d54c>] do_sys_open+0x12c/0x220
Aug 19 05:42:32 HOSTNAME kernel: [496830.814573] [<ffffffff8120d65e>] SyS_open+0x1e/0x20
Aug 19 05:42:32 HOSTNAME kernel: [496830.814576] [<ffffffff816f74f6>] system_call_fastpath+0x18/0xee
Aug 19 05:42:32 HOSTNAME kernel: [496830.814578] INFO: task scsi_id:19785 blocked for more than 120 seconds.
Aug 19 05:42:32 HOSTNAME kernel: [496830.814579] Tainted: P O 4.1.12-124.41.1.bug31517449.el6uek.rc1.x86_64 #2
Aug 19 05:42:32 HOSTNAME kernel: [496830.814579] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug 19 05:42:32 HOSTNAME kernel: [496830.814580] scsi_id D ffff88bebeb58480 0 19785 1 0x00000000
Aug 19 05:42:32 HOSTNAME kernel: [496830.814581] ffff88130d31ba08 0000000000000086 ffff88751e78c600 ffff88751e78c600
Aug 19 05:42:32 HOSTNAME kernel: [496830.814583] ffff88751e78c600 ffff88130d31c000 ffff88be85592124 ffff88751e78c600
Aug 19 05:42:32 HOSTNAME kernel: [496830.814584] 00000000ffffffff 0000000000000000 ffff88130d31ba28 ffffffff816f1de7
Aug 19 05:42:32 HOSTNAME kernel: [496830.814585] Call Trace:
Aug 19 05:42:32 HOSTNAME kernel: [496830.814586] [<ffffffff816f1de7>] schedule+0x37/0x90
Aug 19 05:42:32 HOSTNAME kernel: [496830.814587] [<ffffffff816f216e>] schedule_preempt_disabled+0xe/0x10
Aug 19 05:42:32 HOSTNAME kernel: [496830.814588] [<ffffffff816f43a5>] __mutex_lock_slowpath+0xa5/0x200
Aug 19 05:42:32 HOSTNAME kernel: [496830.814590] [<ffffffff81482288>] ? kobj_lookup+0x118/0x180
Aug 19 05:42:32 HOSTNAME kernel: [496830.814591] [<ffffffff816f4523>] mutex_lock+0x23/0x40
Aug 19 05:42:32 HOSTNAME kernel: [496830.814592] [<ffffffff81310965>] disk_block_events+0x35/0x90
Aug 19 05:42:32 HOSTNAME kernel: [496830.814595] [<ffffffff81249eeb>] __blkdev_get+0x5b/0x490
Aug 19 05:42:32 HOSTNAME kernel: [496830.814596] [<ffffffff8124a35d>] blkdev_get+0x3d/0x380
Aug 19 05:42:32 HOSTNAME kernel: [496830.814597] [<ffffffff8124a75f>] blkdev_open+0x5f/0x90
Aug 19 05:42:32 HOSTNAME kernel: [496830.814599] [<ffffffff8120bcba>] do_dentry_open+0x1ea/0x340
Aug 19 05:42:32 HOSTNAME kernel: [496830.814600] [<ffffffff8124a700>] ? blkdev_get_by_dev+0x60/0x60
Aug 19 05:42:32 HOSTNAME kernel: [496830.814602] [<ffffffff8120d187>] vfs_open+0x57/0x80
Aug 19 05:42:32 HOSTNAME kernel: [496830.814603] [<ffffffff812192f8>] ? may_open+0x68/0x110
Aug 19 05:42:32 HOSTNAME kernel: [496830.814604] [<ffffffff8121c3ca>] do_last+0x21a/0x1210
Aug 19 05:42:32 HOSTNAME kernel: [496830.814605] [<ffffffff8121eda0>] path_openat+0x80/0x650
Aug 19 05:42:32 HOSTNAME kernel: [496830.814606] [<ffffffff81220d5a>] do_filp_open+0x3a/0xb0
Aug 19 05:42:32 HOSTNAME kernel: [496830.814607] [<ffffffff8122ddc7>] ? __alloc_fd+0xa7/0x130
Aug 19 05:42:32 HOSTNAME kernel: [496830.814608] [<ffffffff8120d54c>] do_sys_open+0x12c/0x220
Aug 19 05:42:32 HOSTNAME kernel: [496830.814609] [<ffffffff8120d65e>] SyS_open+0x1e/0x20
Aug 19 05:42:32 HOSTNAME kernel: [496830.814610] [<ffffffff816f74f6>] system_call_fastpath+0x18/0xee
Aug 19 05:42:35 HOSTNAME kernel: [496834.139706] qla2xxx [0000:37:00.0]-801c:8: Abort command issued nexus=8:2:0 -- 0 2003.
Aug 19 05:42:35 HOSTNAME kernel: [496834.235500] qla2xxx [0000:37:00.0]-801c:8: Abort command issued nexus=8:1:4 -- 0 2003.
Aug 19 05:42:47 HOSTNAME kernel: [496846.164928] qla2xxx [0000:86:00.0]-705e:10: Raw firmware dump ready for read on (10).
Aug 19 05:42:47 HOSTNAME kernel: [496846.167370] qla2xxx [0000:86:00.0]-705d:10: Firmware dump cleared on (10).

 

SOS report temporary log file contains these entries towards the end of the file. 

2020-08-19 05:39:24,688 INFO: [plugin:block] collecting output of 'fdisk -l /dev/sdfz'
2020-08-19 05:39:24,706 INFO: [plugin:block] collecting output of 'udevadm info -ap /sys/block/sdhe'
2020-08-19 05:39:24,868 INFO: [plugin:block] collecting output of 'parted -s /dev/sdhe unit s print'
2020-08-19 05:39:24,889 INFO: [plugin:block] collecting output of 'fdisk -l /dev/sdhe'
2020-08-19 05:39:24,909 INFO: [plugin:block] collecting output of 'udevadm info -ap /sys/block/sdgp'
2020-08-19 05:39:25,072 INFO: [plugin:block] collecting output of 'parted -s /dev/sdgp unit s print'
2020-08-19 05:39:25,093 INFO: [plugin:block] collecting output of 'fdisk -l /dev/sdgp'
2020-08-19 05:39:25,112 INFO: [plugin:block] collecting output of 'udevadm info -ap /sys/block/sdhf'
2020-08-19 05:39:25,275 INFO: [plugin:block] collecting output of 'parted -s /dev/sdhf unit s print'
2020-08-19 05:39:25,295 INFO: [plugin:block] collecting output of 'fdisk -l /dev/sdhf'
2020-08-19 05:39:25,313 INFO: [plugin:block] collecting output of 'udevadm info -ap /sys/block/sdgq'
2020-08-19 05:39:25,475 INFO: [plugin:block] collecting output of 'parted -s /dev/sdgq unit s print'
2020-08-19 05:39:25,496 INFO: [plugin:block] collecting output of 'fdisk -l /dev/sdgq'
2020-08-19 05:39:25,514 INFO: [plugin:block] collecting output of 'udevadm info -ap /sys/block/sdhg'
2020-08-19 05:39:25,675 INFO: [plugin:block] collecting output of 'parted -s /dev/sdhg unit s print'

 

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


In this Document
Symptoms
Cause
Solution
References


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.