ACFS Unmounting with error ORA-15130 (Doc ID 1541330.1)

Last updated on NOVEMBER 04, 2016

Applies to:

Oracle Database - Enterprise Edition - Version 11.2.0.3 and later
Information in this document applies to any platform.

Symptoms

ACFS is unmounting with error ORA-15130. Below are error message from crs and css logs.

From ASm alert we can see below

Sat Mar 09 10:48:11 2013
NOTE: repairing group 8 file 256 extent 702
WARNING: Read Failed. group:8 disk:2 AU:2 offset:4096 size:4096
WARNING: cache failed reading from group=8(TWR8DADMIN_DG) fn=256 indblk=1 count=1 from disk= 2(TWR8DADMIN_D01_5910) kfkist=0x20 status=0x02 file=kfc.c line=11556
ERROR: cache failed to read group=8(TWR8DADMIN_DG) fn=256 indblk=1 from disk(s): 2(TWR8DADMIN_D01_5910)
..
Sat Mar 09 10:48:12 2013
ERROR: -15(asmlib ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so op ioerror error I/O Error)
ERROR: -15(asmlib ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so op ioerror error I/O Error)
..
Sat Mar 09 10:48:13 2013
ERROR: ORA-15130 in COD recovery for diskgroup 8/0x1158e7f8 (TWR8DADMIN_DG)
NOTE: cache deleting context for group TWR8DADMIN_DG 8/0x1158e7f8
ERROR: ORA-15130 thrown in RBAL for group number 8
Errors in file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_rbal_12046.trc:
ORA-15130: diskgroup "TWR8DADMIN_DG" is being dismounted
..
Sat Mar 09 10:50:52 2013                                         <<< CSSD terminated
NOTE: ASMB process exiting, either shutdown is in progress
NOTE: or foreground connected to ASMB was killed.

 

From ocssd.log     <<< CSSD termination is confirmed by ocssd.log
~~
2013-03-09 10:50:50.491: [    CSSD][2165586240]clssscMonitorThreads clssnmvWorkerThread not scheduled for 159350 msecs
2013-03-09 10:50:51.089: [    CSSD][2180823360](:CSSNM00058:)clssnmvDiskCheck: No I/O completions for 200990 ms for voting file ORCL:TWR8DOCR_D01_5001)
2013-03-09 10:50:51.089: [    CSSD][2180823360](:CSSNM00018:)clssnmvDiskCheck: Aborting, 0 of 1 configured voting disks available, need 1
2013-03-09 10:50:51.089: [    CSSD][2180823360]###################################
2013-03-09 10:50:51.089: [    CSSD][2180823360]clssscExit: CSSD aborting from thread clssnmvDiskPingMonitorThread
2013-03-09 10:50:51.089: [    CSSD][2180823360]###################################
2013-03-09 10:50:51.089: [    CSSD][2180823360](:CSSSC00012:)clssscExit: A fatal error occurred and the CSS daemon is terminating abnormally
2013-03-09 10:50:51.089: [    CSSD][2180823360]

From messages log

~~
Mar  9 10:48:10 dog1142 kernel:  rport-2:0-2: blocked FC remote port time out: saving binding
Mar  9 10:48:10 dog1142 kernel: sd 2:0:0:3: Unhandled error code
Mar  9 10:48:10 dog1142 kernel: sd 2:0:0:3: SCSI error: return code = 0x00010000
Mar  9 10:48:10 dog1142 kernel: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK,SUGGEST_OK
Mar  9 10:48:10 dog1142 kernel: device-mapper: multipath: Failing path 65:224.
Mar  9 10:48:10 dog1142 kernel: sd 2:0:0:3: Unhandled error code
Mar  9 10:48:10 dog1142 kernel: sd 2:0:0:3: SCSI error: return code = 0x00010000
Mar  9 10:48:10 dog1142 kernel: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK,SUGGEST_OK
Mar  9 10:48:10 dog1142 kernel: sd 2:0:0:14: Unhandled error code
Mar  9 10:48:10 dog1142 kernel: sd 2:0:0:14: SCSI error: return code = 0x00010000
Mar  9 10:48:10 dog1142 kernel: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK,SUGGEST_OK
Mar  9 10:48:10 dog1142 kernel: ADVMK-00020: A read error was reported to the ASM instance for volume asm/twr8d_admin-346 in diskgroup TWR8DADMIN_DG.
Mar  9 10:48:10 dog1142 kernel: ACFSK-0058: I/O failure 0xc0000185 with device /dev/asm/twr8d_admin-346 during a metadata synch read.
Mar  9 10:48:10 dog1142 kernel: ACFSK-0059: Fenum Unknown. Starting offset: 5887563264 . Length of data transfer: 512 bytes.
Mar  9 10:48:10 dog1142 kernel: ACFSK-0060: Impact: Cluster   Object: MetaData   Operation Context: Read Snapshot?: ? AcfsObjectID: 0 .  Internal ACFS Location: 200 .
Mar  9 10:48:10 dog1142 kernel: ACFSK-0058: I/O failure 0xc0000185 with device /dev/asm/twr8d_admin-346 during a metadata synch read.
Mar  9 10:48:10 dog1142 kernel: ACFSK-0059: Fenum: 0 . Starting offset: 5887563264 . Length of data transfer: 512 bytes.

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms