ORA-15196 while allocation reaches 2Tb even compatible.asm is set to 12.1

(Doc ID 1933068.1)

Last updated on OCTOBER 06, 2014

Applies to:

Oracle Database - Enterprise Edition - Version 12.1.0.1 and later
Information in this document applies to any platform.

Symptoms

 

New diskgroup was created with compatible.asm 12.1.0.0.
While restoring to this diskgroup ,the moment file allocation reaches 2Tb ,ASM diskgroup getting corrupted and
dismounting with ORA-15196 error.

While diskgroup was created ,
.
Wed Jun 11 14:20:55 2014
SQL> CREATE DISKGROUP O25T_BACKUP EXTERNAL REDUNDANCY DISK
'/dev/vx/rdsk/o25t_backup_asm_dg/3pardata3_271_vol' SIZE 2044723M ,
'/dev/vx/rdsk/o25t_backup_asm_dg/3pardata3_272_vol' SIZE 2044723M ATTRIBUTE
.
'compatible.asm'='12.1.0.0.0','compatible.rdbms'='11.2','au_size'='4M' /*
ASMCA */ >>>>>>>>>>>>>diskgroup created with 2T disks
Wed Jun 11 14:20:55 2014
NOTE: Assigning number (13,0) to disk
(/dev/vx/rdsk/o25t_backup_asm_dg/3pardata3_271_vol)
NOTE: Assigning number (13,1) to disk
(/dev/vx/rdsk/o25t_backup_asm_dg/3pardata3_272_vol)
NOTE: initializing header (replicated) on grp 13 disk O25T_BACKUP_0000
NOTE: initializing header (replicated) on grp 13 disk O25T_BACKUP_0001

+ Size was 1.94+ Tb.
.
Thu Jun 12 09:25:51 2014
SUCCESS: ALTER DISKGROUP O25T_BACKUP RESIZE DISK O25T_BACKUP_0000 DISK
O25T_BACKUP_0001 /* EM */
Starting background process ARB0
Thu Jun 12 09:25:51 2014
ARB0 started with pid=60, OS id=45134
NOTE: assigning ARB0 to group 13/0x82998fc7 (O25T_BACKUP) with 1024 parallel
I/Os
NOTE: Attempting voting file refresh on diskgroup O25T_BACKUP
Thu Jun 12 09:27:01 2014
NOTE: requesting all-instance membership refresh for group=13
Thu Jun 12 09:27:01 2014
NOTE: membership refresh pending for group 13/0x82998fc7 (O25T_BACKUP)
Thu Jun 12 09:27:01 2014
GMON querying group 13 at 15604 for pid 15, osid 9205
Thu Jun 12 09:27:01 2014
SUCCESS: refreshed membership for 13/0x82998fc7 (O25T_BACKUP)
NOTE: Attempting voting file refresh on diskgroup O25T_BACKUP
Thu Jun 12 09:28:15 2014
NOTE: stopping process ARB0
Thu Jun 12 09:28:16 2014
SUCCESS: rebalance completed for group 13/0x82998fc7 (O25T_BACKUP)
Thu Jun 12 09:31:58 2014
ERROR: failed to read ACD block gn=13 blk=0
Thu Jun 12 09:31:58 2014
Errors in file /opt/ora/gridOB/diag/asm/+asm/+ASM/trace/+ASM_lgwr_9197.trc:
ORA-15196: invalid ASM block header [kfr.c:10703] [endian_kfbh] [3] [0] [6 !=
1]
WARNING: LGWR failed to read ACDC for diskgroup 13 thread 1
Thu Jun 12 09:31:58 2014
Errors in file /opt/ora/gridOB/diag/asm/+asm/+ASM/trace/+ASM_lgwr_9197.trc:
Thu Jun 12 09:31:58 2014
WARNING: cache read (retry) a corrupt block: group=13(O25T_BACKUP) fn=1 blk=1
disk=0 (O25T_BACKUP_0000) incarn=3916006032 au=10 blk=1 count=1
Thu Jun 12 09:31:58 2014
Errors in file /opt/ora/gridOB/diag/asm/+asm/+ASM/trace/+ASM_ora_19652.trc:
ORA-15196: invalid ASM block header [kfc.c:28346] [endian_kfbh] [1] [1] [73
!= 1]
ORA-15196: invalid ASM block header [kfc.c:28346] [endian_kfbh] [1] [1] [73
!= 1]
NOTE: a corrupted block from group O25T_BACKUP was dumped to
/opt/ora/gridOB/diag/asm/+asm/+ASM/trace/+ASM_ora_19652.trc
ERROR: cache failed to read group=13(O25T_BACKUP) fn=1 blk=1 from disk(s):
0(O25T_BACKUP_0000)
ORA-15196: invalid ASM block header [kfc.c:28346] [endian_kfbh] [1] [1] [73
!= 1]
ORA-15196: invalid ASM block header [kfc.c:28346] [endian_kfbh] [1] [1] [73
!= 1]
.
NOTE: cache initiating offline of disk 0 group O25T_BACKUP
NOTE: process _user19652_+asm (19652) initiating offline of disk 0.3916006032
(O25T_BACKUP_0000) with mask 0x7e in group 13 (O25T_BACKUP) with client
assisting
NOTE: initiating PST update: grp 13 (O25T_BACKUP), dsk = 0/0xe9698290, mask =
0x6a, op = clear
Thu Jun 12 09:31:58 2014
GMON updating disk modes for group 13 at 16730 for pid 54, osid 19652
ERROR: disk 0(O25T_BACKUP_0000) in group 13(O25T_BACKUP) cannot be offlined
because the disk group has external redundancy.
Thu Jun 12 09:31:58 2014
ERROR: too many offline disks in PST (grp 13)
Thu Jun 12 09:31:58 2014
NOTE: cache dismounting (not clean) group 13/0x82998FC7 (O25T_BACKUP)
NOTE: messaging CKPT to quiesce pins Unix process pid: 35225, image:
oraclesehan9022cc.han.telia.se (B000)
Thu Jun 12 09:31:58 2014
NOTE: halting all I/Os to diskgroup 13 (O25T_BACKUP)
Thu Jun 12 09:31:58 2014
NOTE: LGWR doing non-clean dismount of group 13 (O25T_BACKUP) thread 1
NOTE: LGWR sync ABA=14.4118 last written ABA 14.4118
Thu Jun 12 09:31:58 2014
NOTE: cache dismounted group 13/0x82998FC7 (O25T_BACKUP)
Thu Jun 12 09:31:58 2014
SQL> alter diskgroup O25T_BACKUP dismount force /* ASM SERVER:2191101895 */
Errors in file /opt/ora/gridOB/diag/asm/+asm/+ASM/trace/+ASM_ora_19652.trc
(incident=176433):
ORA-15335: ASM metadata corruption detected in disk group 'O25T_BACKUP'
ORA-15130: diskgroup "O25T_BACKUP" is being dismounted
ORA-15066: offlining disk "O25T_BACKUP_0000" in group "O25T_BACKUP" may
result in a data loss
ORA-15196: invalid ASM block header [kfc.c:28346] [endian_kfbh] [1] [1] [73
!= 1]
ORA-15196: invalid ASM block header [kfc.c:28346] [endian_kfbh] [1] [1] [73
!= 1]
Incident details in:
/opt/ora/gridOB/diag/asm/+asm/+ASM/incident/incdir_176433/+ASM_ora_19652_i1764
33.trc
Thu Jun 12 09:31:58 2014
ASM Health Checker found 1 new failures
Thu Jun 12 09:31:58 2014
ASM Health Checker found 1 new failures
Thu Jun 12 09:31:58 2014
ERROR: ORA-15130 in COD recovery for diskgroup 13/0x82998fc7 (O25T_BACKUP)
ERROR: ORA-15130 thrown in RBAL for group number 13
Thu Jun 12 09:31:58 2014
Errors in file /opt/ora/gridOB/diag/asm/+asm/+ASM/trace/+ASM_rbal_9205.trc:
ORA-15130: diskgroup "O25T_BACKUP" is being dismounted
.
.
.
kfbh.endian: 73 ; 0x000: 0x49
kfbh.hard: 0 ; 0x001: 0x00
kfbh.type: 110 ; 0x002: *** Unknown Enum ***
kfbh.datfmt: 0 ; 0x003: 0x00
kfbh.block.blk: 7274614 ; 0x004: blk=7274614
kfbh.block.obj: 6881387 ; 0x008: file=589931
kfbh.check: 6750318 ; 0x00c: 0x0067006e
kfbh.fcn.base: 7929939 ; 0x010: 0x00790053
kfbh.fcn.wrap: 7602291 ; 0x014: 0x00740073
kfbh.spare1: 7143525 ; 0x018: 0x006d0065
kfbh.spare2: 4063266 ; 0x01c: 0x003e0022
kfbtTraverseBlock: Invalid OSM block type 110

ORA-15196: invalid ASM block header [kfc.c:28346] [endian_kfbh] [1] [1] [73
!= 1]
ORA-15196: invalid ASM block header [kfc.c:28346] [endian_kfbh] [1] [1] [73
!= 1]
----- Abridged Call Stack Trace -----
ksedsts()+213<-kfcReadBlk()+1824<-kfcLoad()+6326<-kffbScanNext()+279<-kffbTabl
eCb()+861<-kfgTableCb()+986<-kffilTableCb()+203<-qerfxFetch()+3847<-qersoProce
ssULS()+308<-qersoFetch()+6012<-qerjotFetch()+2147<-rwsfcd()+124<-qersoProcess
ULS()+308<-qersoFetchSimple()+1042
<-qersoFetch()+225<-qerjotFetch()+2385<-rwsfcd()+124<-qersoProcessULS()+308<-q
ersoFetchSimple()+1042<-qersoFetch()+225<-qerjotFetch()+2385<-rwsfcd()+124<-qe
rsoProcessULS()+308<-qersoFetchSimple()+1042<-qersoFetch()+225<-qerjotFetch()+
2385<-rwsfcd()+124<-qersoProcessULS()+308
<-qersoFetchSimple()+1042<-qersoFetch()+225<-qerjotFetch()+2385<-qergsFetch()+
531<-rwsfcd()+124<-qeruaFetch()+348<-qervwFetch()+144<-qergsFetch()+531<-rwsfc
d()+124<-qeruaFetch()+348<-qersoProcessULS()+308<-qersoFetchSimple()+1042<-qer
soFetch()+225<-opifch2()+3148
<-kpoal8()+3713<-opiodr()+1170<-ttcpip()+2660<-opitsk()+1751<-opiino()+888<-op
iodr()+1170<-opidrv()+586<-sou2o()+120<-opimai_real()+151<-ssthrdmain()+392<-m
ain()+222<-__libc_start_main()+253
----- End of Abridged Call Stack Trace -----
NOTE: process _user19652_+asm (19652) initiating offline of disk 0.3916006032
(O25T_BACKUP_0000) with mask 0x7e in group 13 (O25T_BACKUP) with client
assisting
NOTE: initiating PST update: grp 13 (O25T_BACKUP), dsk = 0/0xe9698290, mask =
0x6a, op = clear
kfdp_updateDsk(): 16730
DDE rules only execution for: ORA 15066
.
.
File_name :: +ASM_ora_19652.trc


Disk was becoming CANDIDATE ,though no external writes were taking place .

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms