Kernel Panic - Not Syncing: Watchdog Detected Hard LOCKUP (Doc ID 2270077.1)

Last updated on JULY 06, 2017

Applies to:

Linux OS - Version Oracle Linux 6.7 with Unbreakable Enterprise Kernel [4.1.12] and later
Information in this document applies to any platform.

Symptoms

Kernel warnings such as those below are shown in /var/log/messages. Typically this leads to a crash or fence of a node:

[1438363.700124] WARNING: CPU: 89 PID: 0 at net/ipv4/tcp_input.c:2224 tcp_mark_head_lost+0x29d/0x2b0()
...
[1438363.700265] [<ffffffff8163b64d>] tcp_mark_head_lost+0x29d/0x2b0
[1438363.700269] [<ffffffff81640438>] ? tcp_clean_rtx_queue+0x268/0xb10
[1438363.700271] [<ffffffff8163b6d1>] tcp_update_scoreboard+0x71/0x80
[1438363.700273] [<ffffffff8163febd>] tcp_fastretrans_alert+0x4bd/0x660
[1438363.700275] [<ffffffff81641235>] tcp_ack+0x555/0x830
[1438363.700277] [<ffffffff816429e5>] tcp_rcv_established+0x275/0x710
[1438363.700282] [<ffffffff8164c519>] ? tcp_v4_inbound_md5_hash+0x79/0x1b0
[1438363.700286] [<ffffffff8164e043>] tcp_v4_do_rcv+0x103/0x270
[1438363.700289] [<ffffffff8164ef6a>] tcp_v4_rcv+0x80a/0x810
...

 


[3601223.091787] WARNING: CPU: 16 PID: 120 at lib/list_debug.c:33 __list_add+0xbe/0xd0()
...
[3601223.091941] [<ffffffff810ea0a0>] mod_timer_pinned+0xd0/0x160
[3601223.091947] [<ffffffff81632255>] inet_twsk_schedule+0x45/0x60
[3601223.091951] [<ffffffff8164fce2>] tcp_time_wait+0x222/0x280
[3601223.091953] [<ffffffff81643b27>] tcp_rcv_state_process+0x7b7/0x7f0
[3601223.091955] [<ffffffff8164c519>] ? tcp_v4_inbound_md5_hash+0x79/0x1b0
[3601223.091958] [<ffffffff8164e0b4>] tcp_v4_do_rcv+0x174/0x270
...

 

[4023558.175484] Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 7
...
[4023558.175602] [<ffffffff815ce27c>] sk_reset_timer+0x1c/0x30
[4023558.175604] [<ffffffff816400f3>] tcp_rearm_rto+0x93/0x120
[4023558.175606] [<ffffffff81641348>] tcp_ack+0x668/0x830
[4023558.175608] [<ffffffff816429e5>] tcp_rcv_established+0x275/0x710
[4023558.175612] [<ffffffffa10db318>] ? uncond.67352+0x58/0xfffffffffffff4d9 [ip_tables]
[4023558.175614] [<ffffffff8164c519>] ? tcp_v4_inbound_md5_hash+0x79/0x1b0
[4023558.175616] [<ffffffff8164e043>] tcp_v4_do_rcv+0x103/0x270
[4023558.175618] [<ffffffff8164ef6a>] tcp_v4_rcv+0x80a/0x810
...

These errors can occur alone or in combination.



Changes

This issues has been observed in UEK R4 release 1 ( 4.1.12-37.2.1 ) and earlier.

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms