ASM Operations Very Slow when ASM devices were configured on NFS (Doc ID 1424473.1)

Last updated on OCTOBER 31, 2016

Applies to:

Oracle Server - Enterprise Edition - Version: 11.2.0.2 and later   [Release: 11.2 and later ]
Information in this document applies to any platform.

Symptoms



Operations on ASM 6-node production cluster are very slow.
Operations like adding/removing disks (1 hour +) and queries (5 min +) take too much time.



Hang analyze shows:
===============================================================================

Chains most likely to have caused the hang:
[a] Chain 1 Signature: <not in a wait><='rdbms ipc reply'
Chain 1 Signature Hash: 0x5777580e
[b] Chain 2 Signature: 'Disk file operations I/O'<='rdbms ipc reply'
Chain 2 Signature Hash: 0x7278e935
[c] Chain 3 Signature: 'Disk file operations I/O'<='rdbms ipc reply'
Chain 3 Signature Hash: 0x7278e935

===============================================================================
Chain 1:
-------------------------------------------------------------------------------
Oracle session identified by:
{
instance: 1 (+asm.+asm1)
os id: 23939
process id: 29, oracle@ (TNS V1-V3)

is waiting for 'rdbms ipc reply' with wait info:

and is blocked by
=> Oracle session identified by:

instance: 1 (+asm.+asm1)

os id: 6356
process id: 18, oracle@ (RBAL)
blocking: 1 session

Chain 1 Signature: <not in a wait><='rdbms ipc reply'

Chain 1 Signature Hash: 0x5777580e
-------------------------------------------------------------------------------
Chain 2:
-------------------------------------------------------------------------------
Oracle session identified by:
{
instance: 3 (+asm.+asm3)
os id: 13416
process id: 72, oracle@ (TNS V1-V3)

is waiting for 'rdbms ipc reply' with wait info:


and is blocked by

=> Oracle session identified by:

instance: 3 (+asm.+asm3)

os id: 28113
process id: 18, oracle@ (RBAL)

which is waiting for 'Disk file operations I/O' with wait info:

{
blocking: 1 session
wait history:
* time between current wait and wait #1: 0.000063 sec
1. event: 'enq: RB - contention'
2. event: 'enq: RB - contention'
3. event: 'enq: RB - contention'
}

Chain 2 Signature: 'Disk file operations I/O'<='rdbms ipc reply'
Chain 2 Signature Hash: 0x7278e935
-------------------------------------------------------------------------------

/votediskseco2/prodcl20/nfs_votedisk_prodcl20_* devices are not reported by ASM.(v$asm_disk)


In latest diagnostic files there still are some processes trying to use NFS devices:

---
kfgrp: number: 0/0 type: 0 compat: 0.0.0.0.0 dbcompat:0.0.0.0.0
...
disk: num: 0/7084547265882860120 grp: 0/7084547261966843904 compat: 11.2.0.0.0 dbcompat:11.2.0.0.0
fg: path: /votediskseco2/prodcl20/nfs_votedisk_prodcl20_09 <<<
mnt: X hdr: F mode: i v(--) p(r-) a(-) d(-) sta: N flg: 1000
kfts: 2011/11/01 16:09:19.199000
kfts: 2011/11/01 16:09:22.958000
------




RBAL stack contains skgfr_checknfs_mountoptions function:

ksedsts<-ksdxfstk<-ksdxcb<-sspuser<-__sighandler<-__xstat64<-skgfr_checknfs_mountoptions <<<<<<
<-skgfrdevstat<-skgfrdscdevs<-ksfddscdevs<-kfk_dscv_ufs_init
<-kfk_discover_disks<-kfdDiscoverString<-kfdDiscoverShallow<-kfgbDriver()



Changes


Added  'dummy' devices created on the NFS.
All these devices were later removed from the diskgroup but the devices were not removed from the /votediskseco2/prodcl20 location.


Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms