ASM disk reads fail with changed asm_diskstring parameter - ORA-15080: synchronous I/O operation to a disk failed (Doc ID 1645948.1)

Last updated on FEBRUARY 13, 2017

Applies to:

Oracle Database - Enterprise Edition - Version 10.2.0.2 to 11.2.0.4 [Release 10.2 to 11.2]
Linux x86-64
Linux x86
Generic Linux
Main Issue:
==============
ASM disk reads fail
ASM disks didn't get dismounted since Customer changed asm_diskstring parameter.

Symptoms

In the alert_+ASM.log:
...
Tue Jul 09 18:52:48 2013
NOTE: ASM client mcdc:mcdc disconnected unexpectedly.
NOTE: check client alert log.
NOTE: Trace records dumped in trace file /u01/app/oracle/diag/asm/+asm/+ASM/trace/+ASM_ora_20307.trc
...
Tue Jul 09 18:52:50 2013
WARNING: Read Failed. group:2 disk:3 AU:3 offset:0 size:4096
WARNING: cache failed reading from group=2(DATA) fn=6 blk=0 count=1 from disk= 3(DATA04) kfkist=0x20 status=0x02 file=kfc.c line=11555
ERROR: cache failed to read group=2(DATA) fn=6 blk=0 from disk(s): 3(DATA04)
ORA-15080: synchronous I/O operation to a disk failed  ---------------->  HERE
NOTE: cache initiating offline of disk 3 group DATA  ---------------->  HERE
NOTE: process _user27925_+asm (27925) initiating offline of disk 3.3971639839 (DATA04) with mask 0x7e in group 2  ---------------->  HERE
WARNING: Disk 3 (DATA04) in group 2 in mode 0x7f is now being taken offline on ASM inst 1
NOTE: initiating PST update: grp = 2, dsk = 3/0xecba6a1f, mask = 0x6a, op = clear
Tue Jul 09 18:52:51 2013
GMON updating disk modes for group 2 at 13 for pid 25, osid 27925
ERROR: Disk 3 cannot be offlined, since diskgroup has external redundancy.
ERROR: too many offline disks in PST (grp 2)
Tue Jul 09 18:52:51 2013
NOTE: cache dismounting (not clean) group 2/0xBC3A9AFC (DATA)
NOTE: messaging CKPT to quiesce pins Unix process pid: 27946, image: oracle@mcdc01.jud.state.ma.us (B000)
Tue Jul 09 18:52:52 2013
NOTE: halting all I/Os to diskgroup 2 (DATA)           ---------------->  HERE
Tue Jul 09 18:52:52 2013
...
SQL> alter diskgroup DATA dismount force /* ASM SERVER */                   ---------------->  HERE
WARNING: Offline of disk 3 (DATA04) in group 2 and mode 0x7f failed on ASM inst 1  ---------------->  HERE
Errors in file /u01/app/oracle/diag/asm/+asm/+ASM/trace/+ASM_ora_27915.trc:
ORA-17503: ksfdopn:2 Failed to open file +DATA/mcdc/spfilemcdc.ora
ORA-15130: diskgroup "DATA" is being dismounted                              ---------------->  HERE
ORA-15066: offlining disk "DATA04" in group "DATA" may result in a data loss  ---------------->  HERE
ORA-15080: synchronous I/O operation to a disk failed                   ---------------->  HERE
Tue Jul 09 18:52:52 2013
...
SUCCESS: diskgroup DATA was dismounted
SUCCESS: alter diskgroup DATA dismount force /* ASM SERVER */  ---------------->  HERE
Tue Jul 09 18:52:52 2013
NOTE: diskgroup resource ora.DATA.dg is offline
ERROR: PST-initiated MANDATORY DISMOUNT of group DATA
Tue Jul 09 18:52:53 2013
SQL> ALTER DISKGROUP DATA MOUNT  /* asm agent *//* {0:4:26} */  ---------------->  HERE
...
WARNING: Read Failed. group:0 disk:10 AU:0 offset:0 size:4096
Errors in file /u01/app/oracle/diag/asm/+asm/+ASM/trace/+ASM_ora_20178.trc:
ORA-27061: waiting for async I/Os failed  ---------------->  HERE
Linux-x86_64 Error: 5: Input/output error  ---------------->  HERE
Additional information: -1    ---------------->  HERE
Additional information: 4096  ---------------->  HERE
WARNING: Read Failed. group:0 disk:0 AU:0 offset:0 size:4096
NOTE: Assigning number (2,0) to disk (ORCL:LG01)
NOTE: Assigning number (2,1) to disk (ORCL:MED02)
NOTE: Assigning number (2,2) to disk (ORCL:SM01)
...
NOTE: Assigning number (2,3) to disk ()
...
NOTE: cache dismounted group 2/0x117A9B02 (DATA)
NOTE: cache ending mount (fail) of group DATA number=2 incarn=0x117a9b02
NOTE: cache deleting context for group DATA 2/0x117a9b02
GMON dismounting group 2 at 19 for pid 19, osid 20178
NOTE: Disk  in mode 0x8 marked for de-assignment
NOTE: Disk  in mode 0x8 marked for de-assignment
NOTE: Disk  in mode 0x8 marked for de-assignment
NOTE: Disk  in mode 0x8 marked for de-assignment
ERROR: diskgroup DATA was not mounted
ORA-15032: not all alterations performed
ORA-15040: diskgroup is incomplete
ORA-15042: ASM disk "3" is missing from group number "2"   ---------------->  HERE
ORA-15080: synchronous I/O operation to a disk failed
ORA-15080: synchronous I/O operation to a disk failed
ERROR: ALTER DISKGROUP DATA MOUNT  /* asm agent *//* {0:4:26} */  ---------------->  HERE
Tue Jul 09 18:53:00 2013
Errors in file /u01/app/oracle/diag/asm/+asm/+ASM/trace/+ASM_ora_28346.trc:
ORA-27061: waiting for async I/Os failed   ---------------->  HERE
Linux-x86_64 Error: 5: Input/output error  ---------------->  HERE
Additional information: -1
Additional information: 4096
WARNING: Read Failed. group:0 disk:0 AU:0 offset:0 size:4096
WARNING: Read Failed. group:0 disk:10 AU:0 offset:0 size:4096
Tue Jul 09 18:53:00 2013
ASM Health Checker found 1 new failures
Tue Jul 09 18:58:04 2013
NOTE: ASMB process exiting due to lack of ASM file activity for 305 seconds
Wed Jul 10 09:14:37 2013
Shutting down instance (immediate)  ---------------->  HERE
...
Wed Jul 10 09:14:45 2013
Instance shutdown complete
Wed Jul 10 09:14:52 2013
...
Wed Jul 10 09:14:56 2013
SQL> ALTER DISKGROUP ALL MOUNT
NOTE: Diskgroups listed in ASM_DISKGROUPS are
ARCH
DATA
TEMP
...
NOTE: cache opening disk 0 of grp 1: ARCH01 label:MED01
NOTE: F1X0 found on disk 0 au 2 fcn 0.0
NOTE: cache mounting (first) external redundancy group 1/0xE022BD39 (ARCH)
...
NOTE: cache opening disk 0 of grp 2: DATA01 label:LG01
NOTE: F1X0 found on disk 0 au 2 fcn 0.166360
NOTE: cache opening disk 1 of grp 2: DATA02 label:MED02
NOTE: cache opening disk 2 of grp 2: DATA03 label:SM01
NOTE: cache opening disk 3 of grp 2: DATA04 label:LG02
NOTE: cache mounting (first) external redundancy group 2/0xE042BD3A (DATA)
...
NOTE: cache opening disk 0 of grp 3: TEMP01 label:MED03
NOTE: F1X0 found on disk 0 au 2 fcn 0.0
NOTE: cache mounting (first) external redundancy group 3/0xE042BD3B (TEMP)
...
SUCCESS: diskgroup ARCH was mounted               --------------->  ARCH was successfully mounted
GMON querying group 2 at 11 for pid 13, osid 25613
NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 2
SUCCESS: diskgroup DATA was mounted                            ---------------> DATA was successfully mounted
GMON querying group 3 at 12 for pid 13, osid 25613
NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 3
SUCCESS: diskgroup TEMP was mounted                --------------->  TEMP was successfully mounted
SUCCESS: ALTER DISKGROUP ALL MOUNT              --------------->  All diskgroups successfully mounted.
...
Wed Jul 10 16:04:41 2013
WARNING: Read Failed. group:2 disk:3 AU:3 offset:0 size:4096
WARNING: cache failed reading from group=2(DATA) fn=6 blk=0 count=1 from disk= 3(DATA04) kfkist=0x20 status=0x02 file=kfc.c line=11555
ERROR: cache failed to read group=2(DATA) fn=6 blk=0 from disk(s): 3(DATA04)
ORA-15080: synchronous I/O operation to a disk failed  ---------------->  HERE
NOTE: cache initiating offline of disk 3 group DATA  ---------------->  HERE
NOTE: process _user28115_+asm (28115) initiating offline of disk 3.3981594077 (DATA04) with mask 0x7e in group 2
WARNING: Disk 3 (DATA04) in group 2 in mode 0x7f is now being taken offline on ASM inst 1
NOTE: initiating PST update: grp = 2, dsk = 3/0xed524ddd, mask = 0x6a, op = clear
Wed Jul 10 16:04:41 2013
GMON updating disk modes for group 2 at 13 for pid 22, osid 28115
ERROR: Disk 3 cannot be offlined, since diskgroup has external redundancy.
...
Wed Jul 10 16:04:41 2013
NOTE: halting all I/Os to diskgroup 2 (DATA)  ---------------->  HERE
...
SQL> alter diskgroup DATA dismount force /* ASM SERVER */  ---------------->  HERE
Wed Jul 10 16:04:41 2013
NOTE: ASM client mcdc:mcdc disconnected unexpectedly.
NOTE: check client alert log.
NOTE: Trace records dumped in trace file /u01/app/oracle/diag/asm/+asm/+ASM/trace/+ASM_ora_26025.trc
WARNING: Read Failed. group:2 disk:3 AU:1 offset:4096 size:4096
WARNING: Read Failed. group:2 disk:3 AU:1 offset:0 size:4096
Wed Jul 10 16:04:42 2013
ERROR: ORA-15130 in COD recovery for diskgroup 2/0xe042bd3a (DATA)
NOTE: cache deleting context for group DATA 2/0xe042bd3a
ERROR: ORA-15130 thrown in RBAL for group number 2
Errors in file /u01/app/oracle/diag/asm/+asm/+ASM/trace/+ASM_rbal_25613.trc:
ORA-15130: diskgroup "DATA" is being dismounted      ---------------->  HERE
ERROR: ORA-15130 in COD recovery for diskgroup 2/0xe042bd3a (DATA)
ERROR: ORA-15130 thrown in RBAL for group number 2
Errors in file /u01/app/oracle/diag/asm/+asm/+ASM/trace/+ASM_rbal_25613.trc:
ORA-15130: diskgroup "" is being dismounted
...
SUCCESS: diskgroup DATA was dismounted
SUCCESS: alter diskgroup DATA dismount force /* ASM SERVER */
ERROR: PST-initiated MANDATORY DISMOUNT of group DATA
...
Wed Jul 10 16:04:49 2013
SQL> ALTER DISKGROUP DATA MOUNT  /* asm agent *//* {0:4:35} */
...
Wed Jul 10 16:04:55 2013
...
NOTE: cache opening disk 0 of grp 2: DATA01 label:LG01
NOTE: F1X0 found on disk 0 au 2 fcn 0.166360
NOTE: cache opening disk 1 of grp 2: DATA02 label:MED02
NOTE: cache opening disk 2 of grp 2: DATA03 label:SM01
NOTE: cache opening disk 3 of grp 2: DATA04 label:LG02
NOTE: cache mounting (first) external redundancy group 2/0x4CC2BD3D (DATA)
...
Wed Jul 10 16:04:55 2013
NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 2
SUCCESS: diskgroup DATA was mounted
SUCCESS: ALTER DISKGROUP DATA MOUNT  /* asm agent *//* {0:4:35} */  ---------------->  HERE
Thu Jul 11 13:36:22 2013
...
********************************************************************************

 

In the ASM HTML:

GRP NAME STATE    TYPE   TOTAL_MB FREE_MB

1   ARCH MOUNTED EXTERN 20000     7328
2   DATA MOUNTED EXTERN 302816     167872
3   TEMP MOUNTED EXTERN 20000     13120

GRP DISK MOUNT HEADER         MODE STATE TOTAL_MB FREE_MB NAME FAILGROUP LABEL PATH

0 0 IGNORED MEMBER ONLINE NORMAL 0 0                                                          /dev/oracleasm/disks/LG02 ------->  HERE
0 1 IGNORED MEMBER ONLINE NORMAL 0 0                                                          /dev/oracleasm/disks/LG01  ------->  HERE
0 2 IGNORED MEMBER ONLINE NORMAL 0 0                                                         /dev/oracleasm/disks/MED01  ------->  HERE
0 3 IGNORED MEMBER ONLINE NORMAL 0 0                                                         /dev/oracleasm/disks/MED02  ------->  HERE
0 4 IGNORED MEMBER ONLINE NORMAL 0 0                                                         /dev/oracleasm/disks/MED03  ------->  HERE
0 5 IGNORED MEMBER ONLINE NORMAL 0 0                                                         /dev/oracleasm/disks/SM01  ------->  HERE
0 6 CLOSED PROVISIONED ONLINE NORMAL 0 0                                                         /dev/oracleasm/disks/SM02
0 7 CLOSED PROVISIONED ONLINE NORMAL 0 0                                                         /dev/oracleasm/disks/SM03
0 8 CLOSED PROVISIONED ONLINE NORMAL 0 0                                                        /dev/oracleasm/disks/SM04
0 15 CLOSED PROVISIONED ONLINE NORMAL 0 0                                                   SM02 ORCL:SM02
0 16 CLOSED PROVISIONED ONLINE NORMAL 0 0                                                     SM03 ORCL:SM03
0 17 CLOSED PROVISIONED ONLINE NORMAL 0 0                                                   SM04 ORCL:SM04
1 0 CACHED MEMBER     ONLINE NORMAL 20000 7328 ARCH01 ARCH01 MED01 ORCL:MED01   ------->  HERE
2 0 CACHED MEMBER   ONLINE NORMAL 99984 54592 DATA01 DATA01 LG01 ORCL:LG01  ------->  HERE
2 1 CACHED MEMBER       ONLINE NORMAL 20000 10848 DATA02 DATA02 MED02 ORCL:MED02  ------->  HERE
2 2 CACHED MEMBER       ONLINE NORMAL 10000 5408 DATA03 DATA03 SM01 ORCL:SM01  ------->  HERE
2 3 CACHED MEMBER     ONLINE NORMAL 172832 97024 DATA04 DATA04 LG02 ORCL:LG02  ------->  HERE
3 0 CACHED MEMBER        ONLINE NORMAL 20000 13120 TEMP01 TEMP01 MED03 ORCL:MED03  ------->  HERE

current asm_diskstring= /dev/oracleasm/disks/*

****************************************************************************************************************************************************

 

In the OS_COMMANDS.txt (kfod):
...
================================================================================
 1:      99999 Mb /dev/oracleasm/disks/LG01                oracle   dba
 2:     172832 Mb /dev/oracleasm/disks/LG02                oracle   dba
 3:      20000 Mb /dev/oracleasm/disks/MED01               oracle   dba
 4:      20000 Mb /dev/oracleasm/disks/MED02               oracle   dba
 5:      20000 Mb /dev/oracleasm/disks/MED03               oracle   dba
 6:      10000 Mb /dev/oracleasm/disks/SM01                oracle   dba
 7:      10000 Mb /dev/oracleasm/disks/SM02                oracle   dba
 8:      10000 Mb /dev/oracleasm/disks/SM03                oracle   dba
 9:       9679 Mb /dev/oracleasm/disks/SM04                oracle   dba
10:      99999 Mb ORCL:LG01                                <unknown> <unknown>
11:     172832 Mb ORCL:LG02                                <unknown> <unknown>
12:      20000 Mb ORCL:MED01                               <unknown> <unknown>
13:      20000 Mb ORCL:MED02                               <unknown> <unknown>
14:      20000 Mb ORCL:MED03                               <unknown> <unknown>
15:      10000 Mb ORCL:SM01                                <unknown> <unknown>
16:      10000 Mb ORCL:SM02                                <unknown> <unknown>
17:      10000 Mb ORCL:SM03                                <unknown> <unknown>
18:       9679 Mb ORCL:SM04                                <unknown> <unknown>
------------------------------------------------------------------------------------

 

Changes

 

ASM disk reads fail
ASM disks didn't get dismounted since Customer changed asm_diskstring parameter.

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms