Fractured blocks found during crash or instance recovery when using EMC Symmetrix storage

(Doc ID 758055.1)

Last updated on FEBRUARY 08, 2017

Applies to:

Oracle Database - Enterprise Edition - Version 7.0.16.0 to 11.2.0.2 [Release 7.0 to 11.2]
Information in this document applies to any platform.
***Checked for relevance on 07-Dec-2016***

Symptoms

This note applies both to single instance and RAC.

In an Oracle and EMC Symmetrix storage environment, in some very extreme cases, if a node crashes due to a hard error, or there is a hardware error during a write, a surviving instance (in the case of RAC) or the same instance (in the case of single instance) may encounter Fractured Block errors upon crash/instance recovery attempts as follows: 

Hex dump of (file 13, block 360283) in trace file
 /opt/oracle/admin/RDFLA/bdump/rdfla2_p005_30539.trc
 Corrupt block relative dba: 0x03457f5b (file 13, block 360283)
 Fractured block found during crash/instance recovery
 Data in bad block:
 type: 6 format: 2 rdba: 0x03457f5b
 last change scn: 0x0009.ec82ba18 seq: 0x1 flg: 0x02
 spare1: 0x0 spare2: 0x0 spare3: 0x0
 consistency value in tail: 0xdb070601
 check value in block header: 0x0
 block checksum disabled


Subsequent attempts to reuse the affected blocks by any instance results in the ORA-1578 block corrupted error as:

Errors in file /opt/oracle/admin/RDFLA/bdump/rdfla1_smon_21631.trc:
ORA-1578: ORACLE data block corrupted (file # 13, block # 360283)


If a hardware failure is involved, the OS on the failed instance/node might show (on console or system log) IO errors leading to the failure similar to:

<date-time> <nodename> kernel: SCSI error : <1 0 0 107> return code = 0x20000
<date-time> <nodename> kernel: end_request: I/O error, dev sdar, sector 45153
<date-time> <nodename> kernel: end_request: I/O error, dev sdar, sector 45161
<date-time> <nodename> kernel: SCSI error : <1 0 0 133> return code = 0x20000




Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms