ASM RBAL too many open file handlers for device mapper LUNS and hitting system limit (Doc ID 1909546.1)

Last updated on OCTOBER 31, 2016

Applies to:

Oracle Database - Enterprise Edition - Version 11.2.0.3 and later
Information in this document applies to any platform.

Symptoms

ASM RBAL process has 64K Open file handles to device mapper LUNS and unable to see new candidate disks for growing disk group on 1st node of 2 node cluster

The ASM rebalance process is getting errors when it tries spawn a new process to check for candidate LUNS which can be added.
1st node cannot see new LUNS, 2nd node can see new LUNS.
For some unknown reason it has grown to the system limit of 65730 and the majority of those handles are to the device mapper LUN’s dm-<n>.  

Details:

[]# ps -ef | grep asm_rbal
oracle   49866     1  0 May11 ?        00:28:12 asm_rbal_+ASM1

[]# lsof -a -p 49866 | wc -l
65730

strace  -fae -o /tmp/rbal_strace.txt -p 49866

tail -f /tmp/rbal_strace.txt | grep "Too many open files"

49866 socketpair(PF_FILE, SOCK_STREAM, 0, 0xb52cc74) = -1 EMFILE (Too many open files)
49866 open("/etc/oracle/olr.loc", O_RDONLY) = -1 EMFILE (Too many open files)
49866 open("/etc/passwd", O_RDONLY|O_CLOEXEC) = -1 EMFILE (Too many open files)
49866 open("/oracle/app/11.2.0/grid/crs/mesg/crsus.msb", O_RDONLY) = -1 EMFILE (Too many open files)
49866 open("/oracle/app/11.2.0/grid/crs/mesg/crsus.msb", O_RDONLY) = -1 EMFILE (Too many open files)
49866 open("/oracle/app/11.2.0/grid/crs/mesg/crsus.msb", O_RDONLY) = -1 EMFILE (Too many open files)
49866 open("/oracle/app/11.2.0/grid/crs/mesg/crsus.msb", O_RDONLY) = -1 EMFILE (Too many open files)
49866 open("/oracle/app/11.2.0/base/diag/asm/+asm/+ASM1/alert/log.xml", O_WRONLY|O_CREAT|O_APPEND, 0660) = -1 EMFILE (Too many open files)
49866 open("/oracle/app/11.2.0/base/diag/asm/+asm/+ASM1/trace/alert_+ASM1.log", O_WRONLY|O_CREAT|O_APPEND, 0660) = -1 EMFILE (Too many open files)

   KFOD reports

 

kfod asm_diskstring='/dev/mapper/* ' disks=all
--------------------------------------------------------------------------------
Disk          Size Path                                     User     Group  
================================================================================
  1:     204800 Mb /dev/mapper/ASMARCHIVE1               oracle   dba    
  ...
  9:     204800 Mb /dev/mapper/ASMDISK7                     oracle   dba    
 10:     204800 Mb /dev/mapper/ASMDISK8                     oracle   dba    
 11:     204800 Mb /dev/mapper/ASMDISK9                     oracle   dba    
 12:     204813 Mb /dev/mapper/ORAAPPDISK1                root     disk    >>>>>>>>>>
 13:     204800 Mb /dev/mapper/ORAAPPDISK2                root     disk    >>>>>>>>>>
 14:      16384 Mb /dev/mapper/SDVol-LVswap                 root     disk    >>>>>>>>>>
 15:       5120 Mb /dev/mapper/VOTEOCRDISK1               oracle   dba    
 16:       5120 Mb /dev/mapper/VOTEOCRDISK2               oracle   dba    
 17:       5120 Mb /dev/mapper/VOTEOCRDISK3               oracle   dba    
 18:     429215 Mb /dev/mapper/mpatha                         root     disk    >>>>>>>>>>
 19:     429086 Mb /dev/mapper/mpathap2                     root     disk    >>>>>>>>>>
--------------------------------------------------------------------------------


ALso  v$asm_disk shows some disks as CANDIDATE/UNUSED device

Unused Device                  /dev/mapper/ORAAPPDISK1                 0 ONLINE  CLOSED  CANDIDATE
Unused Device                  /dev/mapper/ASMDISK7                    0 ONLINE  CLOSED  CANDIDATE
Unused Device                  /dev/mapper/SDVol-LVswap                0 ONLINE  CLOSED  CANDIDATE
Unused Device                  /dev/mapper/ASMDISK9                    0 ONLINE  CLOSED  CANDIDATE
Unused Device                  /dev/mapper/mpathap2                    0 ONLINE  CLOSED  CANDIDATE
Unused Device                  /dev/mapper/mpatha                      0 ONLINE  CLOSED  CANDIDATE
Unused Device                  /dev/mapper/ORAAPPDISK2                 0 ONLINE  CLOSED  CANDIDATE
Unused Device                  /dev/mapper/ASMDISK8                    0 ONLINE  CLOSED  CANDIDATE

  

lsof output shows a huge number of file descriptors for a non-ASM device

oracle  49866 oracle  266u   BLK              253,7       0t0      9268 /dev/dm-7
oracle  49866 oracle  271r   BLK             253,22    0t8704     31253 /dev/dm-22 >>5457  FD s


lrwxrwxrwx. 1 root root      7 Jun 19 12:14 /dev/mapper/mpatha -> ../dm-7
lrwxrwxrwx. 1 root root      8 Jun 19 01:46 /dev/mapper/SDVol-LVvar_log -> ../dm-22 >>non-ASM device

  

Changes

 Adding new disks for ASM usage

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms