RMAN Merge Recovery Failed to read backuppiece ORA-19566 (Doc ID 1933629.1)

Last updated on FEBRUARY 17, 2019

Applies to:

Goal

Executing RMAN merged incremental backups.  
The backups are being done with RMAN to EMC Data Domain using DDBoost but this is not restricted to Data Domain.  
The backups complete successfully without errors, but the RECOVER COPY WITH TAG is failing with corruption messages reading from the backuppiece.

Error:

ORA-19566: exceeded limit of 0 corrupt blocks for file '....'

Using event 10466 level 1 we are able to test the backup output is true and correct.  Any errors on the channel the backup will fail.

  run
   {
     allocate channel c1 type disk;
     sql channel c1
       "alter session set events ''10466 trace name context forever, level 1''";
     backup database;
   }

There are no errors during the backup only during the merge doing recover when reading the backups from the NFS mounted Data Domain.

RMAN Error:

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on c1 channel at 09/17/2013 11:14:32
ORA-19566: exceeded limit of 0 corrupt blocks for file /<path>/<datafile_copy_name>

Channel debug (krb_trace) level 1 shows:

krbrpr - At reread; tries=1 x->blkcnt_krbrx=1019391 corruption_type=4
krbrrerd; bcktodsk=1 bno=1019391 hardbp=1 x->blkcnt_krbrx=1019391
krbrrerd - Re-reading block from piece
Corrupt block 1019391 found during reading backup piece,
file=/<path>/<backuppiece_name>, corr_type=4
Dump corrupted block
2B8AAF93A000 0000A202 00936EFF 3BEC3CF8 04030817  [.....n...<.;....]
2B8AAF93A010 0000C9D2 0007002B 035E9F75 030393B4  [....+...u.^.....]
2B8AAF93A020 1FE80000 1D681DC0 6C091D14 0007003C  [......h....l<...]
2B8AAF93A030 00DA3075 0090F085 00017ED5 0792C000  [u0.......~......]
2B8AAF93A040 26898EEB 00180204 010EFD04 00821339  [...&........9...]
2B8AAF93A050 003525D2 0792A000 2706CA0F 0002003C  [.%5........'<...]
2B8AAF93A060 00DA3B4A 0090F083 003D7ED5 0792A000  [J;.......~=.....]
2B8AAF93A070 2704269F 00000000 00000000 00000000  [.&.'............]
Reread of blocknum=1019391,
file=//<path>/<backuppiece_name>. found different
data
krbrrerd - exiting
Datafile tail reconstruction error, expected tail of 808714752, got
1022886400 for datafile 2
Operation: 6, isSFT: 0, lastbno: 0, blkcnt: 1019391, rdba: 0x00936eff, tail:
0x30340294, bno_sav: 1273599, hbno: 127, bno: 1273492, newtail: 0x3cf80203

Alert.log shows:

Datafile tail reconstruction error, expected tail of 808714752, got 1022886400 for datafile 2
Corrupt block 1019391 found during reading backup piece, file=/<path>/<backuppiece_name>, corr_type=4
Reread of blocknum=1019391, file=/<path>/<backuppiece_name>. found different data
Datafile tail reconstruction error, expected tail of 808714752, got 1022886400 for datafile 2
No other copies of piece /<path>/<backuppiece_name> to read, reading current piece until end.
Datafile rdba reconstruction error, expected block greater than 1289878, got 1273494 for datafile 2
Corrupt block 1019393 found during reading backup piece, file=/<path>/<backuppiece_name>, corr_type=4
Reread of blocknum=1019393, file=/<path>/<backuppiece_name>. found different data

Solution

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.