My Oracle Support Banner

ODA : Old Disk Path information still exist in ASM (Doc ID 1637898.1)

Last updated on JULY 17, 2019

Applies to:

Oracle Database Appliance - Version All Versions and later
Oracle Database Appliance X5-2
Information in this document applies to any platform.

Symptoms

After a disk replacement in ODA machine , old disk information still exist in environment.

The OLD disk information may exist on one node but not the other.
Either node can see the mismatch.
Sometimes the problem is MISSING information for the new disk.

Regardless: you can get one or more of the following symptoms for the old disk information on one or both nodes.

 

1. Output  of v$asm_disk will still show old disk path with group_number=0 .    < group 0 means the disk belongs to an unowned diskgroup. meaning <does not belong to... Data,Reco,Flash or the Redo diskgroup

2. vgs output will still show IO errors for old disk.

3. /etc/multipath.conf will show old disk information.

4. /dev/mapper/* may show some or all of the old disk information.

5. /dev will also show device information.

6. /dev/mapper may have partial information or multiple entries for partitions for the disk slot.      e.g. You may see partition p2 for the disk but not p1. Also, possibly multiple entries p1 and / or p2 for the same slot

7. /var/log/message  will also record IO errors.

8. ASM alert logs  also display IO errors.

9. /opt/oracle/extapi/asmappl.config may not have references to the disk partition(s) or not seen by one of the enclosures.

 Along with  entries for the old disk , new disk information can also available and it may co-exist for the same diskgroups.

 

*****These symptoms will be on either or both the nodes.*****

 

Example :  Multiple disk references for the same
  Slot (03) 2875 vs. 9843

------------

Check in /dev/mapper to check old information -->>


[root@NODE1 mapper]# ls -altr *S03*
brw-rw---- 1 grid asmadmin 253, 24 Mar 27  2013 HDD_E1_S03_1211732875
brw-rw---- 1 grid asmadmin 253, 30 Mar  7 13:23 HDD_E1_S03_1211732875p1
brw-rw---- 1 grid asmadmin 253, 31 Mar  7 13:30 HDD_E1_S03_1211732875p2
brw-rw---- 1 grid asmadmin 253, 72 Mar 10 14:19 HDD_E1_S03_1797259843
brw-rw---- 1 grid asmadmin 253, 74 Mar 10 16:29 HDD_E1_S03_1797259843p2
brw-rw---- 1 grid asmadmin 253, 73 Mar 10 16:29 HDD_E1_S03_1797259843p1

 

Disk pd_03 was replaced in this environment but ls output  from /dev/mapper is still showing old and new disk information.

PATH HDD_E1_S03_1211732875 is for old disk and PATH HDD_E1_S03_1797259843 is representing new disk.
- Check the Data and Timestamps for another possibly indicator of age confirming the older vs. newer disk(s).

 

Output of "vgs" will show IO errors like given below -->>
Notice that the errors are for the older disk name ..2875

[root@NODE1 mapper]# vgs
  /dev/mpath/HDD_E1_S03_1211732875: read failed after 0 of 4096 at 600127176704: Input/output error
  /dev/mpath/HDD_E1_S03_1211732875: read failed after 0 of 4096 at 600127258624: Input/output error
  /dev/mpath/HDD_E1_S03_1211732875: read failed after 0 of 4096 at 0: Input/output error
  /dev/mpath/HDD_E1_S03_1211732875: read failed after 0 of 4096 at 4096: Input/output error
  /dev/mpath/HDD_E1_S03_1211732875p1: read failed after 0 of 4096 at 515396009984: Input/output error
  /dev/mpath/HDD_E1_S03_1211732875p1: read failed after 0 of 4096 at 515396067328: Input/output error
  /dev/mpath/HDD_E1_S03_1211732875p1: read failed after 0 of 4096 at 0: Input/output error
  /dev/mpath/HDD_E1_S03_1211732875p1: read failed after 0 of 4096 at 4096: Input/output error
  /dev/mpath/HDD_E1_S03_1211732875p2: read failed after 0 of 512 at 84726382592: Input/output error
  /dev/mpath/HDD_E1_S03_1211732875p2: read failed after 0 of 512 at 84726472704: Input/output error
  /dev/mpath/HDD_E1_S03_1211732875p2: read failed after 0 of 512 at 0: Input/output error
  /dev/mpath/HDD_E1_S03_1211732875p2: read failed after 0 of 512 at 4096: Input/output error
  /dev/mpath/HDD_E1_S03_1211732875p2: read failed after 0 of 2048 at 0: Input/output error
  VG          #PV #LV #SN Attr   VSize   VFree
  VolGroupSys   1   4   0 wz--n- 465.66G 251.66G

 

These error messages are related with old disk.

 

ASM alert log error message :--

Mon Mar 10 13:59:39 2014
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_pz99_29322.trc:
ORA-27061: waiting for async I/Os failed
Linux-x86_64 Error: 5: Input/output error
Additional information: -1
Additional information: 4096
WARNING: Read Failed. group:0 disk:51 AU:0 offset:0 size:4096
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_pz99_29322.trc:
ORA-27061: waiting for async I/Os failed
Linux-x86_64 Error: 5: Input/output error
Additional information: -1
Additional information: 4096
ORA-15080: synchronous I/O operation to a disk failed

 

V$ASM_DISK output :--

SQL> select path, name, header_status, mode_status, mount_status, state, failgroup, group_number from v$asm_disk order by path;

PATH                                                               NAME                   HEADER_STATUS      MODE_ST    MOUNT_S       STATE            FAILGROUP                                  GROUP_NUMBER
-------------------------------------                    -----------------           ----------------       ----------    ------------     ---------          ----------------                            ------------------------ ------------
/HDD_E0_S19_1130281880p1                 HDD_E0_S19_1130281880P1      MEMBER          ONLINE       CACHED         NORMAL        HDD_E0_S19_1130281880P1                     2   << --- New disk already added back to diskgroup 2
/HDD_E0_S19_1230211442p1                 HDD_E0_S19_1230211442p1      MEMBER          ONLINE       CLOSED         NORMAL                                                                     0   << --- Group #0 (Old disk now in no group)
 

***Where Group 0 means the disk does not belong to any of the expected disk groups.   

Changes

Failed disk is replaced. 

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.