BDA 4.7 kernel may hang/panic on boot - WARNING: CPU: 16 PID: 4334 At Kernel/workqueue.c (Doc ID 2249602.1)

Last updated on MARCH 30, 2017

Applies to:

Linux OS - Version Oracle Linux 6.0 and later
Linux x86-64

Symptoms

After upgrading to Oracle Big Data Appliance version 4.7, servers may experience on-boot kernel hang/panic with console messages such as the following:
...
  31 [ 123.584706] ------------[ cut here ]------------
  32 [ 123.584711] WARNING: CPU: 16 PID: 4334 at kernel/workqueue.c:1450 __queue_delayed_work+0x13f/0x1b0()
  33 [ 123.584731] Modules linked in: mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_core iTCO_wdt iTCO_vendor_support pcspkr sb_edac edac_core ses enclosure i2c_i801 i2c_core cdc_ether usbnet mii sg lpc_ich mfd_core ioatdma ixgbe dca ptp pps_core vxlan udp_tunnel ip6_udp_tunnel mdio ipmi_devintf ipmi_si ipmi_msghandler ext4 jbd2 mbcache raid1 sd_mod usb_storage ahci libahci megaraid_sas wmi dm_mirror dm_region_hash dm_log dm_mod
  34 [ 123.584733] CPU: 16 PID: 4334 Comm: connectx_port_c Tainted: G W 4.1.12-70.el6uek.x86_64 #2
  35 [ 123.584734] Hardware name: Oracle Corporation ORACLE SERVER X5-2L/ASM,MOBO TRAY,2U, BIOS 31020800 11/17/2014
  36 [ 123.584737] 0000000000000000 ffff881ff06a7c68 ffffffff816c6580 0000000000000000
  37 [ 123.584739] 00000000000005aa ffff881ff06a7ca8 ffffffff810845e5 ffff881ff06a7ce0
  38 [ 123.584741] 0000000000002000 ffff881feb3cfe00 ffff883fe9b82610 ffff883fe9b86d98
  39 [ 123.584742] Call Trace:
  40 [ 123.584745] [] dump_stack+0x63/0x83
  41 [ 123.584749] [] warn_slowpath_common+0x95/0xe0
  42 [ 123.584752] [] warn_slowpath_null+0x1a/0x20
  43 [ 123.584755] [] __queue_delayed_work+0x13f/0x1b0
  44 [ 123.584759] [] queue_delayed_work_on+0x2b/0x50
  45 [ 123.584772] [] mlx4_start_sense+0x3f/0x50 [mlx4_core]
  46 [ 123.584782] [] set_port_type+0x153/0x2e0 [mlx4_core]
  47 [ 123.584784] [] ? __kmalloc+0x1cd/0x2a0
  48 [ 123.584788] [] dev_attr_store+0x20/0x30
  49 [ 123.584791] [] sysfs_kf_write+0x41/0x50
  50 [ 123.584794] [] kernfs_fop_write+0xe9/0x160
  51 [ 123.584797] [] __vfs_write+0x34/0x100
  52 [ 123.584800] [] ? __sb_start_write+0x69/0x100
  53 [ 123.584803] [] ? security_file_permission+0x23/0x90
  54 [ 123.584806] [] vfs_write+0xab/0x120
  55 [ 123.584808] [] SyS_write+0x56/0xd0
  56 [ 123.584810] [] ? do_page_fault+0x37/0x90
  57 [ 123.584813] [] ? syscall_trace_leave+0xf1/0x160
  58 [ 123.584815] [] system_call_fastpath+0x12/0x71
  59 [ 123.584817] ---[ end trace 7fc47b916128df8b ]---
  60 [ 123.584825] ------------[ cut here ]------------
  61 [ 123.584826] kernel BUG at kernel/time/timer.c:974!
  62 [ 123.584829] invalid opcode: 0000 [#1] SMP
  63 [ 123.584848] Modules linked in: mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_core iTCO_wdt iTCO_vendor_support pcspkr sb_edac edac_core ses enclosure i2c_i801 i2c_core cdc_ether usbnet mii sg lpc_ich mfd_core ioatdma ixgbe dca ptp pps_core vxlan udp_tunnel ip6_udp_tunnel mdio ipmi_devintf ipmi_si ipmi_msghandler ext4 jbd2 mbcache raid1 sd_mod usb_storage ahci libahci megaraid_sas wmi dm_mirror dm_region_hash dm_log dm_mod
  64 [ 123.584850] CPU: 16 PID: 4334 Comm: connectx_port_c Tainted: G W 4.1.12-70.el6uek.x86_64 #2
  65 [ 123.584851] Hardware name: Oracle Corporation ORACLE SERVER X5-2L/ASM,MOBO TRAY,2U, BIOS 31020800 11/17/2014
  66 [ 123.584853] task: ffff881fece68e00 ti: ffff881ff06a4000 task.ti: ffff881ff06a4000
  67 [ 123.584858] RIP: 0010:[] [] add_timer_on+0x10b/0x120
  68 [ 123.584859] RSP: 0018:ffff881ff06a7c88 EFLAGS: 00010046
  69 [ 123.584860] RAX: 000000000000eb00 RBX: ffff883fe9b86d38 RCX: 00000000fffcd611
  70 [ 123.584862] RDX: ffff881fff800000 RSI: 0000000000000010 RDI: ffff883fe9b86d38
  71 [ 123.584863] RBP: ffff881ff06a7cb8 R08: 0000000000000000 R09: ffffffff81e9dac0
  72 [ 123.584864] R10: 00000000000dd710 R11: 0000000000000000 R12: ffff881fff80eb40
  73 [ 123.584865] R13: ffff883fe9b86d38 R14: ffff883fe9b86d98 R15: ffff881ff06a7d58
  74 [ 123.584867] FS: 00007f51a1f00700(0000) GS:ffff881fff800000(0000) knlGS:0000000000000000
  75 [ 123.584868] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  76 [ 123.584869] CR2: 00007f51a1f14000 CR3: 0000001fea109000 CR4: 00000000001406a0
  77 [ 123.584871] Stack:
  78 [ 123.584873] ffff881feb3cfe00 0000000000000010 ffff881feb3cfe00 ffff883fe9b86d38
  79 [ 123.584875] ffff883fe9b86d98 ffff881ff06a7d58 ffff881ff06a7cf8 ffffffff8109e6f0
  80 [ 123.584878] 0000000000000c7a ffff883fe9b86d18 ffff881ff06a7ce8 0000000000000292
  81 [ 123.584878] Call Trace:
  82 [ 123.584882] [] __queue_delayed_work+0x90/0x1b0
  83 [ 123.584886] [] queue_delayed_work_on+0x2b/0x50
  84 [ 123.584897] [] mlx4_start_sense+0x3f/0x50 [mlx4_core]
  85 [ 123.584907] [] set_port_type+0x153/0x2e0 [mlx4_core]
  86 [ 123.584909] [] ? __kmalloc+0x1cd/0x2a0
  87 [ 123.584912] [] dev_attr_store+0x20/0x30
  88 [ 123.584916] [] sysfs_kf_write+0x41/0x50
  89 [ 123.584919] [] kernfs_fop_write+0xe9/0x160
  90 [ 123.584921] [] __vfs_write+0x34/0x100
  91 [ 123.584924] [] ? __sb_start_write+0x69/0x100
  92 [ 123.584927] [] ? security_file_permission+0x23/0x90
  93 [ 123.584930] [] vfs_write+0xab/0x120
  94 [ 123.584932] [] SyS_write+0x56/0xd0
  95 [ 123.584934] [] ? do_page_fault+0x37/0x90
  96 [ 123.584937] [] ? syscall_trace_leave+0xf1/0x160
  97 [ 123.584939] [] system_call_fastpath+0x12/0x71
  98 [ 123.584963] Code: 00 4d 85 ed 74 23 49 8b 4d 00 66 0f 1f 44 00 00 49 8b 7d 08 49 83 c5 10 4c 89 f2 48 89 de ff d1 49 8b 4d 00 48 85 c9 75 e7 eb 81 0b 0f 1f 00 eb fb 48 8b 75 08 e8 d5 f6 ff ff e9 2a ff ff ff
  99 [ 123.584965] RIP [] add_timer_on+0x10b/0x120
  100 [ 123.584966] RSP
  101 [ 123.584968] ---[ end trace 7fc47b916128df8c ]---
  102 [ 123.592209] Kernel panic - not syncing: Fatal exception
  103 [ 123.641280] Kernel Offset: disabled
  104 [ 132.400083] Rebooting in 60 seconds..

Changes

 Servers were upgraded to BDA version 4.7

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms