rootupgrade.sh succeeded on all nodes but activeversion wasn't set to the new version (Doc ID 2109628.1)

Last updated on MARCH 12, 2016

Applies to:

Oracle Database - Enterprise Edition - Version 12.1.0.2 and later
Information in this document applies to any platform.

Symptoms

Upgrading 4-node 11.2.0.4 to 12.1.0.2, rootupgrade.sh succeeded on all nodes:

Node1:

2016-01-10 08:14:30: Checking parameters from paramfile /u01/app/12.1.0.2/grid/crs/install/crsconfig_params to validate installer variables
....
2016-01-10 08:23:44: CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded

Node2:

2016-01-10 08:24:15: Checking parameters from paramfile /u01/app/12.1.0.2/grid/crs/install/crsconfig_params to validate installer variables
....
2016-01-10 08:28:46: CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded

Node3:

2016-01-10 08:24:20: Checking parameters from paramfile /u01/app/12.1.0.2/grid/crs/install/crsconfig_params to validate installer variables
....
2016-01-10 08:30:34: CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded

Node4:

2016-01-10 08:31:52: Checking parameters from paramfile /u01/app/12.1.0.2/grid/crs/install/crsconfig_params to validate installer variables
....
2016-01-10 08:36:19: CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded

But activeversion wasn't set the the new version:

# /u01/app/12.1.0.2/grid/bin/crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [11.2.0.4.0]

Checking root script trace on last node:  

2016-01-10 08:32:30: Detect isLastNodeToUpgrade before bringing down the old stack
2016-01-10 08:32:30: leftVersion=11.2.0.4.0; rightVersion=12.1.0.0.0
2016-01-10 08:32:30: [11.2.0.4.0] is lower than [12.1.0.0.0]
2016-01-10 08:32:30: isLastNodeToUpgrade...
....
2016-01-10 08:32:30: [11.2.0.4.0] is lower than [12.1.0.0.0]
2016-01-10 08:32:30: Executing cmd: /u01/app/11.2.0.4/grid/bin/crsctl query crs softwareversion dshee402001por ====>>>> failure getting software version for node1
2016-01-10 08:32:30: Command output:
> Failure 3 in clsvswversion for node dshee402001por
>End Command output
2016-01-10 08:32:30: /u01/app/11.2.0.4/grid/bin/crsctl query crs softwareversion dshee402001por ... failed rc=14 with message:
Failure 3 in clsvswversion for node dshee402001por

2016-01-10 08:32:30: Getting software version for dshee402001por from OCR
2016-01-10 08:32:30: leftVersion=11.2.0.4.0; rightVersion=11.2.0.0.0
2016-01-10 08:32:30: [11.2.0.4.0] is higher than [11.2.0.0.0]
2016-01-10 08:32:30: Getting crs software verison from ocrdump
2016-01-10 08:32:30: ocrdump output for software version on dshee402001por is PROT-310: Not all keys were dumped due to permissions.

2016-01-10 08:32:30: key is
2016-01-10 08:32:30: softver is
2016-01-10 08:32:30: software version retreived from OCR is
2016-01-10 08:32:30: The software version on dshee402001por is
2016-01-10 08:32:30: software_version on dshee402001por=
2016-01-10 08:32:30: last node to upgrade is 0 ====>>>> this is last node but last node flag is NOT set
....
....
2016-01-10 08:32:35: Executing cmd: /u01/app/11.2.0.4/grid/bin/crsctl check crs
2016-01-10 08:32:35: Command output:
> CRS-4638: Oracle High Availability Services is online
> CRS-4537: Cluster Ready Services is online
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
>End Command output
2016-01-10 08:32:35: Earlier version Oracle Clusterware is running

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms