Exadata compute node hangs after reboot with task blocked messages (Doc ID 1352806.1)

Last updated on MAY 24, 2012

Applies to:

Oracle Exadata Hardware - Version 11.2.0.2 to 11.2.0.2 [Release 11.2]
Information in this document applies to any platform.

Symptoms

The compute node is rebooted gracefully but does not come back up following an extended period. The kernel messages, retrieved from a generated ILOM snapshot, may show errors similar to the following:

 

INFO: task oradism:20238 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
oradism D ffffffff80150575 0 20238 1 879 7044 (L-TLB)
ffff8107e62b3c58 0000000000000046 ffff81151f9c2240 ffff81149a4b9610
ffff81151f9c2240 000000000000000a ffff81075d9c67a0 ffff810155df8100
0001ded720678543 00000000000806ac ffff81075d9c6988 000000059a4b9610
Call Trace:
[<ffffffff8877678e>] :nfs:nfs_put_super+0x72/0x88
[<ffffffff800a0b15>] autoremove_wake_function+0x0/0x2e
[<ffffffff80064c7d>] lock_kernel+0x1b/0x32
[<ffffffff800e3bce>] generic_shutdown_super+0x79/0xfb
[<ffffffff800e3c9e>] kill_anon_super+0x9/0x35
[<ffffffff88776913>] :nfs:nfs_kill_super+0xd/0x16


And:


NFO: task kexec:3753 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kexec D ffffffff80150575 0 3753 879 (NOTLB)
ffff8114634afee8 0000000000000082 ffff8114634afe48 ffffffff8004a321
0000000000000000 0000000000000006 ffff81187d874080 ffff810155e07080
0001dedcee326966 000000000000befc ffff81187d874268 00000006634afeb8
Call Trace:
[<ffffffff8004a321>] wait_on_page_writeback_range+0xd6/0x12e
[<ffffffff800646ac>] __down_read+0x7a/0x92
[<ffffffff800f3002>] __sync_inodes+0x4b/0xaa
[<ffffffff800f3072>] sync_inodes+0x11/0x29
[<ffffffff800e0d49>] do_sync+0x12/0x5a


NFS related functions in the kernel error log such as nfs_kill_super and :nfs:nfs_put_super are a good indication that you're hitting this issue. Another requirement is the presence of NFS volumes mounted on the database node, which is playing the role of an NFS client. The remote NFS server may be connected via the Infiniband network.

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms