Offline Disk Failures Caused Many/Duplicated DISMOUNT FORCE to Continue Dismounting Disk Groups (Doc ID 1417701.1)

Last updated on MARCH 16, 2012

Applies to:

Oracle Server - Enterprise Edition - Version: 11.2.0.2 to 11.2.0.3 - Release: 11.2 to 11.2
Information in this document applies to any platform.

Symptoms

If a disk goes offline resulting in a force dismount, kfdp will message a B slave to do this. However, that may be queued behind other messages or there may be a flood of offline requests, all resulting in force dismounts.

There is no need to issue more than one force dismount though since force dismount should never fail. The processes will just be serialized, preventing the other B slaves from doing useful work.

The event sequences were as follows.

1) The ASM instance was up

Sun Aug 28 03:50:25 2011
Starting ORACLE instance (normal)

2) Many offline failures were observed and these generated many force dismount messages to B-slave.

Sat Nov 12 06:32:47 2011
....
WARNING: Write Failed. group:9 disk:18 AU:5 offset:0 size:4096
NOTE: unable to write any mirror side for diskgroup DISK_GROUP_9
NOTE: cache initiating offline of disk 18 group DISK_GROUP_9
ERROR: too many offline disks in PST (grp 9)
WARNING: Offline for disk DISK_GROUP_9_D19 in mode 0x7f failed.
NOTE: halting all I/Os to diskgroup 9 (DISK_GROUP_9)
....
ERROR: too many offline disks in PST (grp 10)
NOTE: halting all I/Os to diskgroup 10 (DISK_GROUP_10)

3) Disk groups 9 and 10 were dismounted. However thereafter, many duplicate dismounts were performed on this ASM instance.

Sat Nov 12 06:32:50 2011
WARNING: dirty detached from domain 10
SQL> alter diskgroup DISK_GROUP_10 dismount force /* ASM SERVER */
SUCCESS: diskgroup DISK_GROUP_10 was dismounted
....
WARNING: dirty detached from domain 9
NOTE: cache dismounted group 9/0x781490F2 (DISK_GROUP_9)
NOTE: cache deleting context for group DISK_GROUP_9 9/0x781490f2
SQL> alter diskgroup DISK_GROUP_9 dismount force /* ASM SERVER */
SQL> alter diskgroup DISK_GROUP_9 dismount force /* ASM SERVER */
SQL> alter diskgroup DISK_GROUP_9 dismount force /* ASM SERVER */
Sat Nov 12 06:32:52 2011
SQL> alter diskgroup DISK_GROUP_9 dismount force /* ASM SERVER */
SUCCESS: diskgroup DISK_GROUP_9 was dismounted

4) Disk groups 9 and 10 were mounted again.

Sun Nov 13 03:08:10 2011
SQL> ALTER DISKGROUP DISK_GROUP_9 MOUNT /* asm agent */
SUCCESS: diskgroup DISK_GROUP_9 was mounted

Sun Nov 13 03:08:22 2011
SQL> ALTER DISKGROUP DISK_GROUP_10 MOUNT /* asm agent */
SUCCESS: diskgroup DISK_GROUP_10 was mounted

5) However the stale dismount messages still existed which continue to dismount the disk groups.

Mon Nov 14 18:12:38 2011
NOTE: cache ignoring dismount request for stale group 9/0x781490f2
SQL> alter diskgroup DISK_GROUP_9 dismount force /* ASM SERVER */
Mon Nov 14 18:12:38 2011
NOTE: cache ignoring dismount request for stale group 10/0xc8749110
SQL> alter diskgroup DISK_GROUP_10 dismount force /* ASM SERVER */
Mon Nov 14 18:12:38 2011
NOTE: cache ignoring dismount request for stale group 9/0x781490f2
SQL> alter diskgroup DISK_GROUP_9 dismount force /* ASM SERVER */
Mon Nov 14 18:12:38 2011
NOTE: cache ignoring dismount request for stale group 10/0xc8749110
SQL> alter diskgroup DISK_GROUP_10 dismount force /* ASM SERVER */



Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms