ORA-15041: diskgroup "DATA" space exhausted during Rebalance (Doc ID 2088372.1)

Last updated on DECEMBER 15, 2015

Applies to:

Oracle Database - Enterprise Edition - Version 11.2.0.4 to 12.1.0.2 [Release 11.2 to 12.1]
Information in this document applies to any platform.

Symptoms

ORA-15041 was reported in ASM1 alert log after 2 grid disks are force dropped from HIGH redundancy diskgroup +DATA.

SQL> select NAME,MOUNT_STATUS,MODE_STATUS,STATE,PATH from V$ASM_DISK;
 .
 NAME                    MOUNT_STATUS MODE_STATUS STATE
 ----------------------- ------------ ----------- --------
 DATA_CD_06_TESTS12M1    MISSING      OFFLINE     NORMAL
DATA_CD_08_TESTS12M1    MISSING      OFFLINE     NORMAL
 DBFS_DG_CD_06_TESTS12M1 MISSING      OFFLINE     NORMAL
 DBFS_DG_CD_08_TESTS12M1 MISSING      OFFLINE     NORMAL
 .
 force dropped the grid disks
 SQL> alter diskgroup DATA drop disk DATA_CD_06_tests12m1 force;
 SQL> alter diskgroup DBFS_DG drop disk DBFS_DG_CD_06_tests12m1 force;
 SQL> alter diskgroup DATA drop disk DATA_CD_08_tests12m1 force;
 SQL> alter diskgroup DBFS_DG drop disk DBFS_DG_CD_08_tests12m1 force;
 SQL> ALTER DISKGROUP DBFS_DG REBALANCE POWER 12 NOWAIT;
 SQL> ALTER DISKGROUP DATA REBALANCE POWER 12 NOWAIT;
 
  Confirmed rebalance was in progress.

 SQL> select * from gv$asm_operation;
 
    INST_ID GROUP_NUMBER OPERATION STATE POWER ACTUAL SOFAR EST_WORK  
 EST_RATE EST_MINUTES
 ---------- ------------ --------------- ------------ ---------- ----------
 ---------- ----
ERROR_CODE
 ----------------------------------------------------------------
  2         2            REBAL WAIT      12
  1         1            REBAL RUN       1   1  5039  1311518       2386    
 547
  1         2            REBAL WAIT      12

  Rebalance stopped and ORA-15041 is reported.
 SQL> select * from gv$asm_operation;
 
    INST_ID GROUP_NUMBER OPERATION STATE    POWER     ACTUAL SOFAR EST_WORK  
 EST_RATE EST_MINUTES
 ---------- ------------ --------------- ------------ ---------- ----------
 ---------- ---------- ---------- -----------
 ERROR_CODE
--------------------------
  2         1             REBAL  ERRS       12
ORA-15041
 1         1             REBAL  ERRS       12
 ORA-15041
 
 
  ASM alert log message
 alert_+ASM1.log
 --------------------
 Fri Oct 02 06:40:27 2015
 ERROR: ORA-15041 thrown in ARB0 for group number 1
 Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_44515.trc:
 ORA-15041: diskgroup "DATA" space exhausted
 Fri Oct 02 06:40:27 2015
NOTE: stopping process ARB0
 NOTE: rebalance interrupted for group 1/0x475d4834 (DATA)

[07:20:39 grid@dpdbs11m1 ~]$ asmcmd lsdg data
 State    Type  Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
 MOUNTED  HIGH  Y         512   4096  4194304  18406784  3463748          1082752          793665              2             N  DATA/

 SQL> select name,failgroup,total_mb,free_mb,hot_used_mb,cold_used_mb  from v$asm_disk order by failgroup,name;
 .
 NAME                 FAILGROUP TOTAL_MB FREE_MB HOT_USED_MB COLD_USED_MB
 -------------------- --------- -------- ------- ----------- ------------
 DATA_CD_00_TESTS12M1 TESTS12M1   541376       0              0          541376
 DATA_CD_01_TESTS12M1 TESTS12M1   541376  103036           0       438340
 DATA_CD_02_TESTS12M1 TESTS12M1   541376   59244           0       482132
 DATA_CD_03_TESTS12M1 TESTS12M1   541376   73596           0       467780
 DATA_CD_04_TESTS12M1 TESTS12M1   541376   78028           0       463348
 DATA_CD_05_TESTS12M1 TESTS12M1   541376   80320           0       461056
 DATA_CD_07_TESTS12M1 TESTS12M1   541376   77876           0       463500
 DATA_CD_09_TESTS12M1 TESTS12M1   541376   40536           0       500840
 DATA_CD_10_TESTS12M1 TESTS12M1   541376   79956           0       461420
 DATA_CD_11_TESTS12M1 TESTS12M1   541376   25780           0       515596

there is only 3 cells and the diskgroup is HIGH redundancy,the data in the 2 dropped disks are relocated to disks in the  same cell server

3 failgroup is used, TESTS11M1, TESTS12M1, TESTS13M1 For diskgroup DATA, originally 12 disks are allocated for each failgroup,disks have same size, about 500G.

ORA-15041 happen when  2 disks of DATA diskgroup is dropped force, two disks belong to failgroup TESTS12M1  Other two disks are dropped from different disk group DBFS_DG, problem does not happen on it, simply that disk group is almost empty

Rebalance was interrupted since it could not find free space on disk 12 which is DATA_CD_00_TESTS12M1. Since no free space on DATA_CD_00_TESTS12M1, rebalance finally failed with ORA-15041.

From GMON trace, the diskgroup DATA is not in APPLIANCE.MODE. Hence, it follows generic partnering(Applies to both Exadata and non-Exadata).

 +ASM1_gmon_117872.trc
 partnering pattern: [ ] <===

 Disk numbers of dropped disks:
 ==========================
 disk 20 of grp 1: DATA_CD_08_TESTS12M1
  disk 18 of grp 1: DATA_CD_06_TESTS12M1

No space on:
=============

 disk 12 of grp 1: DATA_CD_00_TESTS12M1

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms