ASM RBAL Too Many Open File Handlers For Device Mapper LUNS and Hitting System Limit
(Doc ID 1909546.1)
Last updated on DECEMBER 06, 2022
Applies to:
Oracle Database - Enterprise Edition - Version 11.2.0.3 and laterOracle Database Cloud Schema Service - Version N/A and later
Oracle Database Exadata Cloud Machine - Version N/A and later
Oracle Cloud Infrastructure - Database Service - Version N/A and later
Oracle Database Backup Service - Version N/A and later
Information in this document applies to any platform.
Symptoms
ASM RBAL process has 64K Open file handles to device mapper LUNS and unable to see new candidate disks for growing disk group on 1st node of 2 node cluster
The ASM rebalance process is getting errors when it tries spawn a new process to check for candidate LUNS which can be added.
1st node cannot see new LUNS, 2nd node can see new LUNS.
For some unknown reason it has grown to the system limit of 65730 and the majority of those handles are to the device mapper LUN’s dm-<n>.
Details:
oracle 49866 1 0 May11 ? 00:28:12 asm_rbal_+ASM1
[]# lsof -a -p 49866 | wc -l
65730
strace -fae -o /tmp/rbal_strace.txt -p 49866
tail -f /tmp/rbal_strace.txt | grep "Too many open files"
49866 socketpair(PF_FILE, SOCK_STREAM, 0, 0xb52cc74) = -1 EMFILE (Too many open files)
49866 open("/etc/oracle/olr.loc", O_RDONLY) = -1 EMFILE (Too many open files)
49866 open("/etc/passwd", O_RDONLY|O_CLOEXEC) = -1 EMFILE (Too many open files)
49866 open("/oracle/app/11.2.0/grid/crs/mesg/crsus.msb", O_RDONLY) = -1 EMFILE (Too many open files)
49866 open("/oracle/app/11.2.0/grid/crs/mesg/crsus.msb", O_RDONLY) = -1 EMFILE (Too many open files)
49866 open("/oracle/app/11.2.0/grid/crs/mesg/crsus.msb", O_RDONLY) = -1 EMFILE (Too many open files)
49866 open("/oracle/app/11.2.0/grid/crs/mesg/crsus.msb", O_RDONLY) = -1 EMFILE (Too many open files)
49866 open("/oracle/app/11.2.0/base/diag/asm/+asm/+ASM1/alert/log.xml", O_WRONLY|O_CREAT|O_APPEND, 0660) = -1 EMFILE (Too many open files)
49866 open("/oracle/app/11.2.0/base/diag/asm/+asm/+ASM1/trace/alert_+ASM1.log", O_WRONLY|O_CREAT|O_APPEND, 0660) = -1 EMFILE (Too many open files)
KFOD reports
--------------------------------------------------------------------------------
Disk Size Path User Group
================================================================================
1: 204800 Mb /dev/mapper/ASMARCHIVE1 oracle dba
...
9: 204800 Mb /dev/mapper/ASMDISK7 oracle dba
10: 204800 Mb /dev/mapper/ASMDISK8 oracle dba
11: 204800 Mb /dev/mapper/ASMDISK9 oracle dba
12: 204813 Mb /dev/mapper/ORAAPPDISK1 root disk >>>>>>>>>>
13: 204800 Mb /dev/mapper/ORAAPPDISK2 root disk >>>>>>>>>>
14: 16384 Mb /dev/mapper/SDVol-LVswap root disk >>>>>>>>>>
15: 5120 Mb /dev/mapper/VOTEOCRDISK1 oracle dba
16: 5120 Mb /dev/mapper/VOTEOCRDISK2 oracle dba
17: 5120 Mb /dev/mapper/VOTEOCRDISK3 oracle dba
18: 429215 Mb /dev/mapper/mpatha root disk >>>>>>>>>>
19: 429086 Mb /dev/mapper/mpathap2 root disk >>>>>>>>>>
--------------------------------------------------------------------------------
ALso v$asm_disk shows some disks as CANDIDATE/UNUSED device
Unused Device /dev/mapper/ASMDISK7 0 ONLINE CLOSED CANDIDATE
Unused Device /dev/mapper/SDVol-LVswap 0 ONLINE CLOSED CANDIDATE
Unused Device /dev/mapper/ASMDISK9 0 ONLINE CLOSED CANDIDATE
Unused Device /dev/mapper/mpathap2 0 ONLINE CLOSED CANDIDATE
Unused Device /dev/mapper/mpatha 0 ONLINE CLOSED CANDIDATE
Unused Device /dev/mapper/ORAAPPDISK2 0 ONLINE CLOSED CANDIDATE
Unused Device /dev/mapper/ASMDISK8 0 ONLINE CLOSED CANDIDATE
lsof output shows a huge number of file descriptors for a non-ASM device
oracle 49866 oracle 271r BLK 253,22 0t8704 31253 /dev/dm-22 >>5457 FD s
lrwxrwxrwx. 1 root root 7 Jun 19 12:14 /dev/mapper/mpatha -> ../dm-7
lrwxrwxrwx. 1 root root 8 Jun 19 01:46 /dev/mapper/SDVol-LVvar_log -> ../dm-22 >>non-ASM device
Changes
Adding new disks for ASM usage
Cause
To view full details, sign in with your My Oracle Support account. |
|
Don't have a My Oracle Support account? Click to get started! |
In this Document
| Symptoms |
| Changes |
| Cause |
| Solution |
| References |