ACFS Unmounting with error ORA-15130
(Doc ID 1541330.1)
Last updated on DECEMBER 17, 2019
Applies to:
Oracle Database - Enterprise Edition - Version 11.2.0.3 and laterOracle Database Cloud Schema Service - Version N/A and later
Oracle Database Exadata Express Cloud Service - Version N/A and later
Oracle Database Exadata Cloud Machine - Version N/A and later
Oracle Cloud Infrastructure - Database Service - Version N/A and later
Information in this document applies to any platform.
Symptoms
ACFS is unmounting with error ORA-15130. Below are error message from crs and css logs.
From ASm alert we can see below
Sat Mar 09 10:48:11 2013
NOTE: repairing group 8 file 256 extent 702
WARNING: Read Failed. group:8 disk:2 AU:2 offset:4096 size:4096
WARNING: cache failed reading from group=8(<DGNAME>) fn=256 indblk=1 count=1 from disk= 2(<DISK #10>) kfkist=0x20 status=0x02 file=kfc.c line=11556
ERROR: cache failed to read group=8(<DGNAME>) fn=256 indblk=1 from disk(s): 2(<DISK #10>)
..
Sat Mar 09 10:48:12 2013
ERROR: -15(asmlib ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so op ioerror error I/O Error)
ERROR: -15(asmlib ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so op ioerror error I/O Error)
..
Sat Mar 09 10:48:13 2013
ERROR: ORA-15130 in COD recovery for diskgroup 8/0x1158e7f8 (<DGNAME>)
NOTE: cache deleting context for group <DGNAME> 8/0x1158e7f8
ERROR: ORA-15130 thrown in RBAL for group number 8
Errors in file <ORACLE_BASE>/diag/asm/+asm/+ASM2/trace/+ASM2_rbal_12046.trc:
ORA-15130: diskgroup "<DGNAME>" is being dismounted
..
Sat Mar 09 10:50:52 2013 <<< CSSD terminated
NOTE: ASMB process exiting, either shutdown is in progress
NOTE: or foreground connected to ASMB was killed.
NOTE: repairing group 8 file 256 extent 702
WARNING: Read Failed. group:8 disk:2 AU:2 offset:4096 size:4096
WARNING: cache failed reading from group=8(<DGNAME>) fn=256 indblk=1 count=1 from disk= 2(<DISK #10>) kfkist=0x20 status=0x02 file=kfc.c line=11556
ERROR: cache failed to read group=8(<DGNAME>) fn=256 indblk=1 from disk(s): 2(<DISK #10>)
..
Sat Mar 09 10:48:12 2013
ERROR: -15(asmlib ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so op ioerror error I/O Error)
ERROR: -15(asmlib ASM:/opt/oracle/extapi/64/asm/orcl/1/libasm.so op ioerror error I/O Error)
..
Sat Mar 09 10:48:13 2013
ERROR: ORA-15130 in COD recovery for diskgroup 8/0x1158e7f8 (<DGNAME>)
NOTE: cache deleting context for group <DGNAME> 8/0x1158e7f8
ERROR: ORA-15130 thrown in RBAL for group number 8
Errors in file <ORACLE_BASE>/diag/asm/+asm/+ASM2/trace/+ASM2_rbal_12046.trc:
ORA-15130: diskgroup "<DGNAME>" is being dismounted
..
Sat Mar 09 10:50:52 2013 <<< CSSD terminated
NOTE: ASMB process exiting, either shutdown is in progress
NOTE: or foreground connected to ASMB was killed.
From ocssd.log <<< CSSD termination is confirmed by ocssd.log
~~
2013-03-09 10:50:50.491: [ CSSD][2165586240]clssscMonitorThreads clssnmvWorkerThread not scheduled for 159350 msecs
2013-03-09 10:50:51.089: [ CSSD][2180823360](:CSSNM00058:)clssnmvDiskCheck: No I/O completions for 200990 ms for voting file ORCL:<DISK #11>)
2013-03-09 10:50:51.089: [ CSSD][2180823360](:CSSNM00018:)clssnmvDiskCheck: Aborting, 0 of 1 configured voting disks available, need 1
2013-03-09 10:50:51.089: [ CSSD][2180823360]###################################
2013-03-09 10:50:51.089: [ CSSD][2180823360]clssscExit: CSSD aborting from thread clssnmvDiskPingMonitorThread
2013-03-09 10:50:51.089: [ CSSD][2180823360]###################################
2013-03-09 10:50:51.089: [ CSSD][2180823360](:CSSSC00012:)clssscExit: A fatal error occurred and the CSS daemon is terminating abnormally
2013-03-09 10:50:51.089: [ CSSD][2180823360]
2013-03-09 10:50:51.089: [ CSSD][2180823360](:CSSNM00058:)clssnmvDiskCheck: No I/O completions for 200990 ms for voting file ORCL:<DISK #11>)
2013-03-09 10:50:51.089: [ CSSD][2180823360](:CSSNM00018:)clssnmvDiskCheck: Aborting, 0 of 1 configured voting disks available, need 1
2013-03-09 10:50:51.089: [ CSSD][2180823360]###################################
2013-03-09 10:50:51.089: [ CSSD][2180823360]clssscExit: CSSD aborting from thread clssnmvDiskPingMonitorThread
2013-03-09 10:50:51.089: [ CSSD][2180823360]###################################
2013-03-09 10:50:51.089: [ CSSD][2180823360](:CSSSC00012:)clssscExit: A fatal error occurred and the CSS daemon is terminating abnormally
2013-03-09 10:50:51.089: [ CSSD][2180823360]
From messages log
~~
Mar 9 10:48:10 <HOSTNAME> kernel: rport-2:0-2: blocked FC remote port time out: saving binding
Mar 9 10:48:10 <HOSTNAME> kernel: sd 2:0:0:3: Unhandled error code
Mar 9 10:48:10 <HOSTNAME> kernel: sd 2:0:0:3: SCSI error: return code = 0x00010000
Mar 9 10:48:10 <HOSTNAME> kernel: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK,SUGGEST_OK
Mar 9 10:48:10 <HOSTNAME> kernel: device-mapper: multipath: Failing path 65:224.
Mar 9 10:48:10 <HOSTNAME> kernel: sd 2:0:0:3: Unhandled error code
Mar 9 10:48:10 <HOSTNAME> kernel: sd 2:0:0:3: SCSI error: return code = 0x00010000
Mar 9 10:48:10 <HOSTNAME> kernel: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK,SUGGEST_OK
Mar 9 10:48:10 <HOSTNAME> kernel: sd 2:0:0:14: Unhandled error code
Mar 9 10:48:10 <HOSTNAME> kernel: sd 2:0:0:14: SCSI error: return code = 0x00010000
Mar 9 10:48:10 <HOSTNAME> kernel: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK,SUGGEST_OK
Mar 9 10:48:10 <HOSTNAME> kernel: ADVMK-00020: A read error was reported to the ASM instance for volume asm/<DEVICE_NAME> in diskgroup <DGNAME>.
Mar 9 10:48:10 <HOSTNAME> kernel: ACFSK-0058: I/O failure 0xc0000185 with device /dev/asm/<DEVICE_NAME> during a metadata synch read.
Mar 9 10:48:10 <HOSTNAME> kernel: ACFSK-0059: Fenum Unknown. Starting offset: 5887563264 . Length of data transfer: 512 bytes.
Mar 9 10:48:10 <HOSTNAME> kernel: ACFSK-0060: Impact: Cluster Object: MetaData Operation Context: Read Snapshot?: ? AcfsObjectID: 0 . Internal ACFS Location: 200 .
Mar 9 10:48:10 <HOSTNAME> kernel: ACFSK-0058: I/O failure 0xc0000185 with device /dev/asm/<DEVICE_NAME> during a metadata synch read.
Mar 9 10:48:10 <HOSTNAME> kernel: ACFSK-0059: Fenum: 0 . Starting offset: 5887563264 . Length of data transfer: 512 bytes.
Mar 9 10:48:10 <HOSTNAME> kernel: sd 2:0:0:3: Unhandled error code
Mar 9 10:48:10 <HOSTNAME> kernel: sd 2:0:0:3: SCSI error: return code = 0x00010000
Mar 9 10:48:10 <HOSTNAME> kernel: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK,SUGGEST_OK
Mar 9 10:48:10 <HOSTNAME> kernel: device-mapper: multipath: Failing path 65:224.
Mar 9 10:48:10 <HOSTNAME> kernel: sd 2:0:0:3: Unhandled error code
Mar 9 10:48:10 <HOSTNAME> kernel: sd 2:0:0:3: SCSI error: return code = 0x00010000
Mar 9 10:48:10 <HOSTNAME> kernel: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK,SUGGEST_OK
Mar 9 10:48:10 <HOSTNAME> kernel: sd 2:0:0:14: Unhandled error code
Mar 9 10:48:10 <HOSTNAME> kernel: sd 2:0:0:14: SCSI error: return code = 0x00010000
Mar 9 10:48:10 <HOSTNAME> kernel: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK,SUGGEST_OK
Mar 9 10:48:10 <HOSTNAME> kernel: ADVMK-00020: A read error was reported to the ASM instance for volume asm/<DEVICE_NAME> in diskgroup <DGNAME>.
Mar 9 10:48:10 <HOSTNAME> kernel: ACFSK-0058: I/O failure 0xc0000185 with device /dev/asm/<DEVICE_NAME> during a metadata synch read.
Mar 9 10:48:10 <HOSTNAME> kernel: ACFSK-0059: Fenum Unknown. Starting offset: 5887563264 . Length of data transfer: 512 bytes.
Mar 9 10:48:10 <HOSTNAME> kernel: ACFSK-0060: Impact: Cluster Object: MetaData Operation Context: Read Snapshot?: ? AcfsObjectID: 0 . Internal ACFS Location: 200 .
Mar 9 10:48:10 <HOSTNAME> kernel: ACFSK-0058: I/O failure 0xc0000185 with device /dev/asm/<DEVICE_NAME> during a metadata synch read.
Mar 9 10:48:10 <HOSTNAME> kernel: ACFSK-0059: Fenum: 0 . Starting offset: 5887563264 . Length of data transfer: 512 bytes.
Changes
Cause
To view full details, sign in with your My Oracle Support account. |
|
Don't have a My Oracle Support account? Click to get started! |
In this Document
| Symptoms |
| Changes |
| Cause |
| Solution |