OCR Corruption and backup failure with "Dev/Page/Block [0/843/1904] is CORRUPT (header)" reported in crsd.log

(Doc ID 1913262.1)

Last updated on AUGUST 29, 2014

Applies to:

Oracle Database - Enterprise Edition - Version 11.2.0.1 and later
Information in this document applies to any platform.

Symptoms

OCR corruption messages are reported in crsd.log, automatic OCR backup is failing. Ocrcheck complains "Device/File intergrity check failed":

[root@racnode1 ~]# ocrcheck
Status of Oracle Cluster Registry is as follows :
        Version                  :          3
        Total space (kbytes)     :     262120
        Used space (kbytes)      :       3372
        Available space (kbytes) :     258748
        ID                       : 1423232882
        Device/File Name         :   +DBFS_DG
                                   Device/File integrity check failed <<<<<<<<<<<<<<<<<<<<<<<<

                                   Device/File not configured

                                   Device/File not configured

                                   Device/File not configured

                                   Device/File not configured

        Cluster registry integrity check failed

        Logical corruption check bypassed due to insufficient quorum

 
<GRID_HOME>/log/racnode1/alertracnode1.log shows:

[crsd(77158)]CRS-1006:The OCR location +DBFS_DG is inaccessible. Details in /u01/app/11.2.0.3/grid/log/racnode1/crsd/crsd.log.
2014-07-28 19:12:18.023:
[/u01/app/11.2.0.3/grid/bin/orarootagent.bin(77413)]CRS-5822:Agent '/u01/app/11.2.0.3/grid/bin/orarootagent_root' disconnected from server. Details at (:CRSAGF00117:) {0:2:6} in /u01/app/11.2.0.3/grid/log/racnode1/agent/crsd/orarootagent_root/orarootagent_root.log.
2014-07-28 19:12:47.718:
[ohasd(40904)]CRS-2765:Resource 'ora.crsd' has failed on server 'racnode1'.
2014-07-28 19:12:54.369:
[crsd(9099)]CRS-1012:The OCR service started on node racnode1.
2014-07-28 19:12:55.702:
[crsd(9099)]CRS-1201:CRSD started on node racnode1.
2014-07-29 03:45:36.471:
[crsd(9099)]CRS-1006:The OCR location +DBFS_DG is inaccessible. Details in /u01/app/11.2.0.3/grid/log/racnode1/crsd/crsd.log.

 
<GRID_HOME>/log/racnode1/crsd/crsd.log shows:

2014-07-31 07:13:09.240: [  OCRRAW][2175183168]proprior:1 Retrying buffer read from another mirror for disk group [+DBFS_DG] for block at offset [7696384]
2014-07-31 07:13:09.244: [  OCRASM][2175183168]proprasmres: Block from mirror #1 is same as buffer passed
2014-07-31 07:13:09.254: [  OCRASM][2175183168]proprasmres: Block from mirror #2 is same as buffer passed
2014-07-31 07:13:09.278: [  OCRASM][2175183168]proprasmres: Total 2 mirrors detected
2014-07-31 07:13:09.278: [  OCRASM][2175183168]proprasmres: Block from mirror #1 same as block from mirror #2
2014-07-31 07:13:09.278: [  OCRASM][2175183168]proprasmres: 2 mirrors found in this disk group.
2014-07-31 07:13:09.278: [  OCRASM][2175183168]proprasmres: The buffer passed matches the buffers read from all 2 mirrors.
2014-07-31 07:13:09.278: [  OCRASM][2175183168]proprasmres: Need to invoke checkdg. The buffer passed matches with buffer from all mirrors.
2014-07-31 07:13:09.488: [  OCRASM][2175183168]proprasmres: Successfully returned after calling Check DG.
2014-07-31 07:13:09.488: [  OCRRAW][2175183168]proprior:1 ASM re silver returned [22]
2014-07-31 07:13:09.488: [  OCRRAW][2175183168]gst: Dev/Page/Block [0/843/1904] is CORRUPT (header)       <<<
2014-07-31 07:13:09.488: [  OCRRAW][2175183168]rbkp:2: Problem [26]. Could not read the free list
2014-07-31 07:13:09.488: [  OCRRAW][2175183168]gst:could not read fcl page 1
2014-07-31 07:13:09.488: [  OCRRAW][2175183168]rbkp:2: Problem [26]. Could not read the free list
2014-07-31 07:13:09.488: [  OCRRAW][2175183168]gst:could not read fcl page 2
2014-07-31 07:13:09.488: [  OCRSRV][2175183168]th_snap:6''':Failed corruption check reading device [+DBFS_DG]. Not taking backup.  <<<
2014-07-31 07:13:09.488: [  OCRSRV][2175183168]th_snap:8:failed to take backup retval [0] corruption [1]
2014-07-31 07:13:36.549: [UiServer][2156271936] CS(0x2628b20)set Properties ( grid,0x7f747c01ff30)

  

Changes

No recent Change

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms