rootupgrade.sh failure: CRS-1111 [Upgrade Coordinator failed] due to existing OCR export

(Doc ID 2121960.1)

Last updated on APRIL 06, 2016

Applies to:

Oracle Database - Enterprise Edition - Version 12.1.0.2 and later
Information in this document applies to any platform.

Symptoms

Upgrading to 12.1.0.2 GI, rootupgrade.sh fails on last node:

root script trace

2016/02/18 14:06:01 CLSRSC-482: Running command: '/u01/app/12.1.0.2/grid/bin/crsctl set crs activeversion'

Started to upgrade the Oracle Clusterware. This operation may take a few minutes.
Started to upgrade the OCR.
CRS-1111: Error upgrading the Oracle Clusterware. Details [Upgrade Coordinator failed].
CRS-1112: Failed to set the Oracle Clusterware operating version 12.1.0.2.0
CRS-4000: Command Set failed, or completed with errors.
2016/02/18 14:06:03 CLSRSC-180: An error occurred while executing the command '/u01/app/12.1.0.2/grid/bin/crsctl set crs activeversion' (error code 1)

2016/02/18 14:06:03 CLSRSC-281: Failed to set active version of Oracle Clusterware

 

CRSD master crashed:

2016-02-18 14:06:02.443242 : OCRSRV:3141506816: th_upgrade:11.1 AV [202375680]. State [1]. Upgrade OCR format. Updated global data to the crs version group. Return [0]
2016-02-18 14:06:02.666325 : OCRAPI:3141506816: proa_upgrade: OCR format version [3] < current OCR format version [4]. Continuing to upgrade OCR.
2016-02-18 14:06:02.666983 : OCRAPI:3141506816: th_upg_oper[20]: Export file [/u01/app/12.1.0.2/grid/cdata/txp3-clu1/ocr12.1.0.2.0_upg_txp3dbadm04.ocr] cannot be deleted.
2016-02-18 14:06:02.666988 : OCRAPI:3141506816: proa_upgrade: Failed to export the OCR by Cache Writer. Retval [14].
2016-02-18 14:06:02.668122 : OCRSRV:3141506816: th_upgrade:12 OCR upgrade failed [14]
2016-02-18 14:06:02.668175 : OCRAPI:3141506816: procr_ctx_set_invalid: ctx is in state [15].
2016-02-18 14:06:02.668179 : OCRAPI:3141506816: procr_ctx_set_invalid: ctx set to invalid
2016-02-18 14:06:02.668255 : OCRAPI:3141506816: procr_ctx_set_invalid: Aborting...
Trace file /u01/app/grid/diag/crs/txp3dbadm04/crs/trace/crsd.trc

 

After that, crsd.trc is flooded with the following:

2016-02-18 15:14:26.391870 : OCRAPI:426868224: a_upgrade_cluster_failure: I am not the first node in the cluster. Releasing the lock and continue monitoring the OCR upgrade state.
2016-02-18 15:14:26.397190 : OCRAPI:426868224: a_upgrade_cluster_failure: OCR format upgrade is in state [15]. Determining if I am the first node in the cluster.
2016-02-18 15:14:26.397978 : OCRAPI:426868224: a_upgrade_cluster_failure: I am not the first node in the cluster. Releasing the lock and continue monitoring the OCR upgrade state.
2016-02-18 15:14:26.405771 : OCRAPI:426868224: a_upgrade_cluster_failure: OCR format upgrade is in state [15]. Determining if I am the first node in the cluster.
2016-02-18 15:14:26.406577 : OCRAPI:426868224: a_upgrade_cluster_failure: I am not the first node in the cluster. Releasing the lock and continue monitoring the OCR upgrade state.
2016-02-18 15:14:26.412060 : OCRAPI:426868224: a_upgrade_cluster_failure: OCR format upgrade is in state [15]. Determining if I am the first node in the cluster.
2016-02-18 15:14:26.412828 : OCRAPI:426868224: a_upgrade_cluster_failure: I am not the first node in the cluster. Releasing the lock and continue monitoring the OCR upgrade state.

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms