Kernel Panic at "net/core/skbuff.c:147!" (Doc ID 1964343.1)

Last updated on JUNE 19, 2017

Applies to:

Linux OS - Version Oracle Linux 5.5 with Unbreakable Enterprise Kernel [2.6.32] and later
Linux x86-64

Symptoms

Server unexpectedly reboots with kernel panic at "net/core/skbuff.c:147!":

KERNEL: vmlinuz-2.6.39-200.24.1.el6uek.x86_64.debug
   DUMPFILE: vmcore  [PARTIAL DUMP]
       CPUS: 32
       DATE: Tue Dec 30 11:20:51 2014
     UPTIME: 46 days, 16:26:27
LOAD AVERAGE: 3.40, 3.01, 3.02
      TASKS: 1417
   NODENAME: ns2sdp1
    RELEASE: 2.6.39-200.24.1.el6uek.x86_64
    VERSION: #1 SMP Sat Jun 23 02:39:07 EDT 2012
    MACHINE: x86_64  (2599 Mhz)
     MEMORY: 128 GB
      PANIC: "kernel BUG at net/core/skbuff.c:147!"
        PID: 6654
    COMMAND: "ocssd.bin"
       TASK: ffff881f764bc5c0  [THREAD_INFO: ffff881981f7e000]
        CPU: 24
      STATE: TASK_RUNNING (PANIC)

crash> bt
PID: 6654   TASK: ffff881f764bc5c0  CPU: 24  COMMAND: "ocssd.bin"
#0 [ffff881981f7f620] machine_kexec at ffffffff810385c9
#1 [ffff881981f7f690] crash_kexec at ffffffff810b5752
#2 [ffff881981f7f760] oops_end at ffffffff81506e18
#3 [ffff881981f7f790] die at ffffffff8101669b
#4 [ffff881981f7f7c0] do_trap at ffffffff81506704
#5 [ffff881981f7f820] do_invalid_op at ffffffff81014605
#6 [ffff881981f7f8c0] invalid_op at ffffffff8150ec3b
   [exception RIP: skb_push+123]
   RIP: ffffffff8142c28b  RSP: ffff881981f7f978  RFLAGS: 00010292
   RAX: 0000000000000087  RBX: ffff881f7b7ee000  RCX: 0000000000000000
   RDX: 0000000000000000  RSI: 0000000000000082  RDI: 0000000000000246
   RBP: ffff881981f7f998   R8: 0000000000000000   R9: ffffffff81559ec0
   R10: 0000000000000003  R11: 0000000000000001  R12: ffff881f7f7f27b0
   R13: 0000000000000000  R14: 0000000000000800  R15: 000000000000010e
   ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
#7 [ffff881981f7f990] cpumask_next_and at ffffffff812539d9
#8 [ffff881981f7f9e0] neigh_resolve_output at ffffffff81444c63
#9 [ffff881981f7fa30] ip_finish_output at ffffffff81475a9e
#10 [ffff881981f7fa80] ip_output at ffffffff81475f80
#11 [ffff881981f7fab0] ip_local_out at ffffffff81474f89
#12 [ffff881981f7fad0] ip_send_skb at ffffffff81474fab
#13 [ffff881981f7faf0] udp_send_skb at ffffffff81498e3a
#14 [ffff881981f7fb40] udp_sendmsg at ffffffff8149a15a
#15 [ffff881981f7fc80] inet_sendmsg at ffffffff814a35c8
#16 [ffff881981f7fcc0] sock_sendmsg at ffffffff814246df
#17 [ffff881981f7fe40] sys_sendto at ffffffff814248a9
#18 [ffff881981f7ff80] system_call_fastpath at ffffffff8150dc82
   RIP: 00000030faa0eba3  RSP: 00007fffe3f77220  RFLAGS: 00000246
   RAX: 000000000000002c  RBX: ffffffff8150dc82  RCX: 00000030faa0e94c
   RDX: 00000000000000e4  RSI: 00007fffb00082a8  RDI: 000000000000004e
   RBP: 00007fffe3f78420   R8: 00007fffd805be68   R9: 0000000000000010
   R10: 0000000000004000  R11: 0000000000000293  R12: 0000000000000003
   R13: 0000000000000000  R14: 00007fffe3f7b9c0  R15: 00007fffffff8270
   ORIG_RAX: 000000000000002c  CS: 0033  SS: 002b

crash> log
...
skb_under_panic: text:ffffffff8145611a len:284 put:14 head:ffff8818298e0800 data:ffff8818298e07f4 tail:0x110 end:0x280 dev:bond1
------------[ cut here ]------------
kernel BUG at net/core/skbuff.c:147!
invalid opcode: 0000 [#1] SMP
CPU 24
Modules linked in: oracleacfs(P)(U) oracleadvm(P)(U) oracleoks(P)(U) oracleasm autofs4 sunrpc pcc_cpufreq bonding ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 ext3 jbd microcode pcspkr serio_raw iTCO_wdt iTCO_vendor_support hpilo hpwdt sg be2net(U) ext4 mbcache jbd2 dm_round_robin scsi_dh_emc sd_mod usb_storage hpsa lpfc(U) scsi_transport_fc scsi_tgt crc_t10dif video dm_multipath dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]

Pid: 6654, comm: ocssd.bin Tainted: P            2.6.39-200.24.1.el6uek.x86_64 #1 HP ProLiant BL460c Gen8
RIP: 0010:[<ffffffff8142c28b>]  [<ffffffff8142c28b>] skb_push+0x7b/0x80
RSP: 0018:ffff881981f7f978  EFLAGS: 00010292
RAX: 0000000000000087 RBX: ffff881f7b7ee000 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000082 RDI: 0000000000000246
RBP: ffff881981f7f998 R08: 0000000000000000 R09: ffffffff81559ec0
R10: 0000000000000003 R11: 0000000000000001 R12: ffff881f7f7f27b0
R13: 0000000000000000 R14: 0000000000000800 R15: 000000000000010e
FS:  00007fffe3f7b700(0000) GS:ffff88203f100000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ffff58eb000 CR3: 0000001f7a545000 CR4: 00000000000406e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process ocssd.bin (pid: 6654, threadinfo ffff881981f7e000, task ffff881f764bc5c0)
Stack:
0000000000000110 0000000000000280 ffff881f7b7ee000 ffffffff812539d9
ffff881981f7f9d8 ffffffff8145611a ffff881981f7fa08 ffff881360b77880
ffff881f7f7f2700 ffff881f7b7ee000 0000000000000fa4 ffff881f7f7f27a8
Call Trace:
[<ffffffff812539d9>] ? cpumask_next_and+0x29/0x50
[<ffffffff8145611a>] eth_header+0x3a/0xe0
[<ffffffff81444c63>] neigh_resolve_output+0x103/0x1d0
[<ffffffff81474060>] ? ip_append_page+0x550/0x550
[<ffffffff81475a9e>] ip_finish_output+0x16e/0x340
[<ffffffff81475f80>] ip_output+0xb0/0xc0
[<ffffffff81474f89>] ip_local_out+0x29/0x30
[<ffffffff81474fab>] ip_send_skb+0x1b/0x70
[<ffffffff81498e3a>] udp_send_skb+0x11a/0x3c0
[<ffffffff81474060>] ? ip_append_page+0x550/0x550
[<ffffffff81474060>] ? ip_append_page+0x550/0x550
[<ffffffff8149a15a>] udp_sendmsg+0x32a/0x8f0
[<ffffffff814d2203>] ? unix_stream_sendmsg+0x423/0x440
[<ffffffff814a35c8>] inet_sendmsg+0x48/0xb0
[<ffffffff814246df>] sock_sendmsg+0xef/0x120
[<ffffffff8125b795>] ? timerqueue_del+0x35/0x90
[<ffffffff81091ae8>] ? hrtimer_try_to_cancel+0x48/0xe0
[<ffffffff811ab9b8>] ? ep_send_events_proc+0xe8/0x110
[<ffffffff8150593e>] ? _raw_spin_lock+0xe/0x20
[<ffffffff81057b4a>] ? task_rq_lock+0x5a/0xb0
[<ffffffff814225c5>] ? copy_from_user+0x35/0x50
[<ffffffff814248a9>] sys_sendto+0x139/0x190
[<ffffffff810cc437>] ? audit_syscall_entry+0x1d7/0x200
[<ffffffff8150dc82>] system_call_fastpath+0x16/0x1b
Code: 89 4c 24 10 8b 8f d0 00 00 00 48 89 4c 24 08 8b bf cc 00 00 00 89 f1 48 8b 75 08 48 89 3c 24 48 c7 c7 38 79 72 81 e8 27 69 0d 00 <0f> 0b eb fe 90 55 48 89 e5 48 83 ec 20 66 66 66 66 90 8b 57 6c
RIP  [<ffffffff8142c28b>] skb_push+0x7b/0x80
RSP <ffff881981f7f978>

  

 

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms