bug 18610307:: deadlock resulting rpc throttle causing node reboot (Doc ID 2030238.1)

Last updated on FEBRUARY 13, 2019

Applies to:

Symptoms

oracle database is down automatically alongwith node.

resulting outage at cluster level.

In messages file,we see ,

May 17 02:46:14 xxxxxxxxxxxx kernel: [Oracle OKS] ASSERTION FAILURE: FALSE File: /scratch/builds/aime/aime_usm_922089/usm/src/oks/driver/./odlmrebuild.c Line: 3218
May 17 02:46:14 xxxxxxxxxxxx kernel: KsDumpStack: Stack call traceback
May 17 02:46:14 xxxxxxxxxxxx kernel: Pid: 26344, comm: acfsutil.bin Tainted: P W --------------- 2.6.32-279.37.2.el6.x86_64 #1
May 17 02:46:14 xxxxxxxxxxxx kernel: Call Trace:
May 17 02:46:14 xxxxxxxxxxxx kernel: [<ffffffffa078abdb>] ? KsDumpStack+0x3b/0x40 [oracleoks]
May 17 02:46:14 xxxxxxxxxxxx kernel: [<ffffffffa078e586>] ? KsDoAssertion+0xd6/0xf0 [oracleoks]
May 17 02:46:14 xxxxxxxxxxxx kernel: [<ffffffffa07a1020>] ? odlm_wait_for_membership_change+0x1d0/0x310 [oracleoks]
May 17 02:46:14 xxxxxxxxxxxx kernel: [<ffffffffa0799779>] ? odlm_wait_for_rbld+0xe9/0x140 [oracleoks]
May 17 02:46:14 xxxxxxxxxxxx kernel: [<ffffffffa07ad1e9>] ? new_lock_remote+0x4e9/0xac0 [oracleoks]
May 17 02:46:14 xxxxxxxxxxxx kernel: [<ffffffffa079e090>] ? create_lock+0x720/0xc60 [oracleoks]
May 17 02:46:14 xxxxxxxxxxxx kernel: [<ffffffffa079ed0f>] ? odlm_lock+0x2af/0x460 [oracleoks]
May 17 02:46:14 xxxxxxxxxxxx kernel: [<ffffffffa088401f>] ? OfsAcquireDLMLock_int+0x175f/0x23b0 [oracleacfs]
May 17 02:46:14 xxxxxxxxxxxx kernel: [<ffffffffa08b6ae0>] ? OfsFreeMemory+0x70/0xb0 [oracleacfs]
May 17 02:46:14 xxxxxxxxxxxx kernel: [<ffffffffa089a21f>] ? OfsDeallocateIOCB+0x5f/0x70 [oracleacfs]
May 17 02:46:14 xxxxxxxxxxxx kernel: [<ffffffffa0941d1a>] ? OfsRecoveryGetLocalDirBlock+0x6a/0xe0 [oracleacfs]
May 17 02:46:14 xxxxxxxxxxxx kernel: [<ffffffffa09424d8>] ? OfsRecoverNode+0x218/0x830 [oracleacfs]
May 17 02:46:14 xxxxxxxxxxxx kernel: [<ffffffffa078cd56>] ? KsRwLockWrite+0x16/0x40 [oracleoks]
May 17 02:46:14 xxxxxxxxxxxx kernel: [<ffffffffa0942f4c>] ? OfsDoPhase1Recovery+0x1cc/0x4d0 [oracleacfs]
May 17 02:46:14 xxxxxxxxxxxx kernel: [<ffffffffa0945c68>] ? OfsWaitForRecoveryToComplete+0x118/0x440 [oracleacfs]
May 17 02:46:14 xxxxxxxxxxxx kernel: [<ffffffffa08836c4>] ? OfsAcquireDLMLock_int+0xe04/0x23b0 [oracleacfs]
May 17 02:46:14 xxxxxxxxxxxx kernel: [<ffffffff81214541>] ? avc_has_perm+0x71/0x90
May 17 02:46:14 xxxxxxxxxxxx kernel: [<ffffffffa0879330>] ? ofs_dlm_blkrtn+0x0/0x170 [oracleacfs]
May 17 02:46:14 xxxxxxxxxxxx kernel: [<ffffffffa0884e23>] ? OfsOpAcquireDLMLock_int+0xc3/0x960 [oracleacfs]
May 17 02:46:14 xxxxxxxxxxxx kernel: [<ffffffffa095fab5>] ? ofs_getattr+0xb5/0x300 [oracleacfs]
May 17 02:46:14 xxxxxxxxxxxx kernel: [<ffffffff8117b971>] ? vfs_getattr+0x51/0x80
May 17 02:46:14 xxxxxxxxxxxx kernel: [<ffffffff8117ba00>] ? vfs_fstatat+0x60/0x80
May 17 02:46:14 xxxxxxxxxxxx kernel: [<ffffffff8117bb4b>] ? vfs_stat+0x1b/0x20
May 17 02:46:14 xxxxxxxxxxxx kernel: [<ffffffff8117bb74>] ? sys_newstat+0x24/0x50
May 17 02:46:14 xxxxxxxxxxxx kernel: [<ffffffff810d3fa7>] ? audit_syscall_entry+0x1d7/0x200
May 17 02:46:14 xxxxxxxxxxxx kernel: [<ffffffff8100b072>] ? system_call_fastpath+0x16/0x1b
May 17 02:46:14 xxxxxxxxxxxx kernel: OKSK-00007: Information has been saved in the file /var/log/Kokslog.0 . Include the contents of this file if reporting a problem to Oracle.
May 17 02:46:14 xxxxxxxxxxxx kernel: OKSK-00007: Information has been saved in the file /var/log/Fokslog.0 . Include the contents of this file if reporting a problem to Oracle.
May 17 02:46:15 xxxxxxxxxxxx kernel: (v 45364) GUARD-02: 31144 khash_super_prune_nolock: SUPER PRUNE at 1431830775 (line 277)
May 17 02:46:19 xxxxxxxxxxxx kernel: (v 45364) GUARD-02: 28292 khash_super_prune_nolock: SUPER PRUNE at 1431830779 (line 277)
May 17 02:46:22 xxxxxxxxxxxx kernel: (v 45364) GUARD-02: 31185 khash_super_prune_nolock: SUPER PRUNE at 1431830782 (line 277)
May 17 02:46:24 xxxxxxxxxxxx kernel: ADVMK-00017: The ASM instance terminated unexpectedly. All ADVM volumes will be taken offline.

File_name :: messages-20150517

Changes

Cause

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.