PRE-11.2 TO 12.1.0.2 UPGRADE FAILED DUE TO ERROR ON STARTING NODEAPPS (Doc ID 1961187.1)

Last updated on APRIL 30, 2015

Applies to:

Oracle Database - Enterprise Edition - Version 10.2.0.5 and later
Information in this document applies to any platform.

Symptoms

Upgrading clusterware from 11107 to 12.1.0.2 ,used force upgrade to finish clusterware upgrade,which failed due to error on starting up the nodeapps.

4 node cluster, switched off node3 and node4. Tried force upgrade on node2 as node3 and node4 was un-available.

Run rootupgrade.sh on first node <host01> successfully.
Power off nodes <host03>/<host04>.
Run rootupgrade.sh on second node <host02> ...

[root@<host02> ~]# date;/u01/app/12.1.0/grid/rootupgrade.sh -force
Fri Aug 23 01:09:41 PDT 2013
Performing root user operation for Oracle 12c.
The following environment variables are set as:
    ORACLE_OWNER= crsusr
    ORACLE_HOME=  /u01/app/12.1.0/grid.
Enter the full pathname of the local bin directory: [/usr/local/bin]:
/usr/local/bin is read only.  Continue without copy (y/n) or retry (r)? [y]:
Warning: /usr/local/bin is read only. No files will be copied.
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file:/u01/app/12.1.0/grid/crs/install/crsconfig_params
2013/08/23 01:09:44 CLSRSC-464: Starting retrieval of the cluster configuration data
2013/08/23 01:09:46 CLSRSC-465: Retrieval of the cluster configuration data has successfully completed.
2013/08/23 01:09:46 CLSRSC-363: User ignored prerequisites during installation
2013/08/23 01:09:56 CLSRSC-472: Attempting to export the OCR
2013/08/23 01:09:56 CLSRSC-482: Command to run: 'ocrconfig -upgrade crsusr oinstall'
2013/08/23 01:09:58 CLSRSC-473: Successfully exported the OCR
2013/08/23 01:09:58 CLSRSC-343: Successfully started Oracle clusterware stack
clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 12c Release 1.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Start upgrade invoked..
2013/08/23 01:10:21 CLSRSC-478: Setting Oracle Clusterware active version on the last node to be upgraded
2013/08/23 01:10:21 CLSRSC-482: Command to run:'/u01/app/12.1.0/grid/bin/crsctl set crs activeversion -force'
Attempting to forcibly upgrade the Oracle Clusterware using only the nodes <host01>, <host02>.
Started to upgrade the Oracle Clusterware. This operation may take a fewminutes.
Started to upgrade the OCR.
Started to upgrade the CSS.
Started to upgrade the CRS.
The CRS was successfully upgraded.
Forcibly upgraded the Oracle Clusterware.
Oracle Clusterware operating version was forcibly set to 12.1.0.2.0
CRS-1121: Oracle Clusterware was forcibly upgraded without upgrading nodes <host03>, <host04>.
2013/08/23 01:12:19 CLSRSC-479: Successfully set Oracle Clusterware active version
2013/08/23 01:12:28 CLSRSC-180: An error occurred while executing the command '/u01/app/12.1.0/grid/bin/srvctl start nodeapps -n <host01>' (error code 1)
2013/08/23 01:12:29 CLSRSC-180: An error occurred while executing the command '/u01/app/12.1.0/grid/bin/srvctl start nodeapps -n <host02>' (error code 1)
2013/08/23 01:12:30 CLSRSC-278: Failed to perform last node configuration for upgrade Died at /u01/app/12.1.0/grid/crs/install/crsupgrade.pm line 3283.
The command '/u01/app/12.1.0/grid/perl/bin/perl -I/u01/app/12.1.0/grid/perl/lib -I/u01/app/12.1.0/grid/crs/install/u01/app/12.1.0/grid/crs/install/rootcrs.pl  -upgrade -force' execution failed
application_racusr.log
---------------------------
2013-08-23 01:11:46.210: [CLSFRAME][1396160896] TM [MultiThread] is changing desired thread # to 10. Current # is 12
2013-08-23 01:11:46.210: [CLSFRAME][1114147136]{0:1:27} Worker thread is exiting in TM [MultiThread] to meet the desired count of 10. New count is 11
2013-08-23 01:11:46.211: [CLSFRAME][1095678272]{0:1:28} Worker thread is exiting in TM [MultiThread] to meet the desired count of 10. New count is 10
2013-08-23 01:12:15.564: [    AGFW][1099438400]{1:47813:304} Agent received the message: AGENT_HB[Engine] ID 12293:1210
2013-08-23 01:12:20.138: [    AGFW][1099438400]{2:7090:1026} Agent received the message: RESOURCE_MODIFY_ATTR[ora.<host01>.ASM1.asm 1 1] ID 4355:1228
2013-08-23 01:12:20.139: [ora.<host01>.ASM1.asm][1107843392]{2:7090:1026} [res_attr_modified] Executing action script:/u01/app/racusr/product/dbhome/bin/racgwrap[res_attr_modified]
2013-08-23 01:12:20.192: [ora.<host01>.ASM1.asm][1107843392]{2:7090:1026} [res_attr_modified] Usage /u01/app/racusr/product/dbhome/bin/racgmain[resource name] start|stop|check
2013-08-23 01:12:20.192: [ora.<host01>.ASM1.asm][1107843392]{2:7090:1026} [res_attr_modified]       /u01/app/racusr/product/dbhome/bin/racgmain startorp|failsrvsa dbname instname [srvname]
2013-08-23 01:12:20.192: [ora.<host01>.ASM1.asm][1107843392]{2:7090:1026} [res_attr_modified]       /u01/app/racusr/product/dbhome/bin/racgmain startorp|failsrvsa nodename
2013-08-23 01:12:20.192: [ora.<host01>.ASM1.asm][1107843392]{2:7090:1026} [res_attr_modified]       /u01/app/racusr/product/dbhome/bin/racgmain cond_resname cond_state func [args...]
2013-08-23 01:12:20.192: [    AGFW][1107843392]{2:7090:1026} Command: res_attr_modified for resource: ora.<host01>.ASM1.asm 1 1 completed with status: FAIL

Changes

Used force upgrade to finish clusterware upgrade.Force root upgrade failed due to error or start node app.

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms