Grid Infrastructure 12c/12.1.0.x rootupgrade.sh fails: PRCR-1065 : Failed to stop resource ora.gsd (Doc ID 1580360.1)

Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
PRCR-1065 : Failed to stop resource ora.gsd
CRS-5013: Agent <GI_HOME>/bin/oraagent.bin" failed to start process <GI_HOME>/bin/gsdctl" for action "clean": details at "(:CLSN00008:)" in <GI_HOME>/log/usfldbh2/agent/crsd/oraagent_oracle/oraagent_oracle.log"
CRS-5013: Agent <GI_HOME>/bin/oraagent.bin" failed to start process <GI_HOME>/bin/gsdctl" for action "check": details at "(:CLSN00008:)" in <GI_HOME>/log/usfldbh2/agent/crsd/oraagent_oracle/oraagent_oracle.log"
CRS-2680: Clean of 'ora.gsd' on 'usfldbh2' failed
2013/08/29 02:48:20 CLSRSC-180: An error occurred while executing the command 'srvctl upgrade model -s 11.2.0.3.0 -d 12.1.0.1.0 -p first' (error code 0)
2013/08/29 02:48:20 CLSRSC-280: Failed to perform first node tasks for cluster modeling upgrade
Died at <GI_HOME>/crs/install/crsupgrade.pm line 3254.
The command <GI_HOME>/perl/bin/perl -I<GI_HOME>/perl/lib -I<GI_HOME>/crs/install <GI_HOME>/crs/install/rootcrs.pl -upgrade -auto' execution failed

2013-08-29 02:47:22: The software version on usfldbh2 is 12.1.0.1.0
2013-08-29 02:47:22: Invoking <GI_HOME>/bin/srvctl upgrade model -s 11.2.0.3.0 -d 12.1.0.1.0 -p first"
2013-08-29 02:47:22: trace file=/opt/oracle/oragrid/12.1.0.1/cfgtoollogs/crsconfig/srvmcfg0.log
2013-08-29 02:47:22: Executing <GI_HOME>/bin/srvctl upgrade model -s 11.2.0.3.0 -d 12.1.0.1.0 -p first
2013-08-29 02:47:22: Executing cmd: <GI_HOME>/bin/srvctl upgrade model -s 11.2.0.3.0 -d 12.1.0.1.0 -p first
2013-08-29 02:48:20: Command output:
> PRCR-1065 : Failed to stop resource ora.gsd
> CRS-5013: Agent <GI_HOME>/bin/oraagent.bin" failed to start process <GI_HOME>/bin/gsdctl" for action "clean": details at "(:CLSN00008:)" in <GI_HOME>/log/usfldbh2/agent/crsd/oraagent_oracle/oraagent_oracle.log"
> CRS-5013: Agent <GI_HOME>/bin/oraagent.bin" failed to start process <GI_HOME>/bin/gsdctl" for action "check": details at "(:CLSN00008:)" in <GI_HOME>/log/usfldbh2/agent/crsd/oraagent_oracle/oraagent_oracle.log"
> CRS-2680: Clean of 'ora.gsd' on 'usfldbh2' failed
>End Command output
2013-08-29 02:48:20: <GI_HOME>/bin/srvctl upgrade model -s 11.2.0.3.0 -d 12.1.0.1.0 -p first" failed with status 1.
2013-08-29 02:48:20: Executing cmd: <GI_HOME>/bin/clsecho -p has -f clsrsc -m 180 "srvctl upgrade model -s 11.2.0.3.0 -d 12.1.0.1.0 -p first" "0"
2013-08-29 02:48:20: Command output:
> CLSRSC-180: An error occurred while executing the command 'srvctl upgrade model -s 11.2.0.3.0 -d 12.1.0.1.0 -p first' (error code 0)
>End Command output
2013-08-29 02:48:20: Executing cmd: <GI_HOME>/bin/clsecho -p has -f clsrsc -m 180 "srvctl upgrade model -s 11.2.0.3.0 -d 12.1.0.1.0 -p first" "0"
2013-08-29 02:48:20: Command output:
> CLSRSC-180: An error occurred while executing the command 'srvctl upgrade model -s 11.2.0.3.0 -d 12.1.0.1.0 -p first' (error code 0)
>End Command output
2013-08-29 02:48:20: CLSRSC-180: An error occurred while executing the command 'srvctl upgrade model -s 11.2.0.3.0 -d 12.1.0.1.0 -p first' (error code 0)
2013-08-29 02:48:20: Running as user oracle: <GI_HOME>/bin/cluutil -ckpt -oraclebase /opt/oracle/orabase -writeckpt -name ROOTCRS_NODECONFIG -state FAIL
2013-08-29 02:48:20: s_run_as_user2: Running /bin/su oracle -c ' <GI_HOME>/bin/cluutil -ckpt -oraclebase /opt/oracle/orabase -writeckpt -name ROOTCRS_NODECONFIG -state FAIL '
2013-08-29 02:48:20: Removing file /tmp/fileVzyfMf
2013-08-29 02:48:20: Successfully removed file: /tmp/fileVzyfMf
2013-08-29 02:48:20: /bin/su successfully executed

2013-08-29 02:48:20: Succeeded in writing the checkpoint:'ROOTCRS_NODECONFIG' with status:FAIL
2013-08-29 02:48:20: Executing cmd: <GI_HOME>/bin/clsecho -p has -f clsrsc -m 280
2013-08-29 02:48:20: Command output:
> CLSRSC-280: Failed to perform first node tasks for cluster modeling upgrade
>End Command output
2013-08-29 02:48:20: Executing cmd: <GI_HOME>/bin/clsecho -p has -f clsrsc -m 280
2013-08-29 02:48:20: Command output:
> CLSRSC-280: Failed to perform first node tasks for cluster modeling upgrade
>End Command output
2013-08-29 02:48:20: CLSRSC-280: Failed to perform first node tasks for cluster modeling upgrade
2013-08-29 02:48:20: ###### Begin DIE Stack Trace ######
2013-08-29 02:48:20: Package File Line Calling
2013-08-29 02:48:20: --------------- -------------------- ---- ----------
2013-08-29 02:48:20: 1: main rootcrs.pl 211 crsutils::dietrap
2013-08-29 02:48:20: 2: crsupgrade crsupgrade.pm 3254 main::__ANON__
2013-08-29 02:48:20: 3: crsupgrade crsupgrade.pm 239 crsupgrade::upgrade_node
2013-08-29 02:48:20: 4: crsupgrade crsupgrade.pm 190 crsupgrade::CRSUpgrade
2013-08-29 02:48:20: 5: main rootcrs.pl 220 crsupgrade::new
2013-08-29 02:48:20: ####### End DIE Stack Trace #######

2013-08-29 02:48:20: ROOTCRS_NODECONFIG checkpoint has failed
2013-08-29 02:48:20: Running as user oracle: <GI_HOME>/bin/cluutil -ckpt -oraclebase /opt/oracle/orabase -writeckpt -name ROOTCRS_NODECONFIG -state FAIL
2013-08-29 02:48:20: s_run_as_user2: Running /bin/su oracle -c ' <GI_HOME>/bin/cluutil -ckpt -oraclebase /opt/oracle/orabase -writeckpt -name ROOTCRS_NODECONFIG -state FAIL '
2013-08-29 02:48:20: Removing file /tmp/filegNLxAd
2013-08-29 02:48:20: Successfully removed file: /tmp/filegNLxAd
2013-08-29 02:48:20: /bin/su successfully executed

2013-08-29 02:46:56.309: [ora.gsd][71309632] {1:3721:2} [check] Utils:execCmd action = 3 flags = 6 ohome = (null) cmdname = gsdctl.
2013-08-29 02:46:56.310: [ora.gsd][71309632] {1:3721:2} [check] getOracleHomeAttrib: oracle_home = <GI_HOME>
2013-08-29 02:46:56.312: [ora.gsd][71309632] {1:3721:2} [check] (:CLSN00008:)Utils:execCmd scls_process_spawn() failed 1
2013-08-29 02:46:56.312: [ora.gsd][71309632] {1:3721:2} [check] (:CLSN00008:) category: -2, operation: stat, loc: procspawn5, OS error: 2, other: cmd <GI_HOME>/bin/gsdctl not found
2013-08-29 02:46:56.313: [ora.gsd][71309632] {1:3721:2} [check] clsnUtils::error Exception type=2 string=
CRS-5013: Agent <GI_HOME>/bin/oraagent.bin" failed to start process <GI_HOME>/bin/gsdctl" for action "check": details at "(:CLSN00008:)" in <GI_HOME>/log/<hostname>/agent/crsd/oraagent_oracle/oraagent_oracle.log"

2013-08-29 02:47:02.432: [ AGFW][3923724608] {1:3721:2} Agent received the message: RESOURCE_START[ora.gsd <hostname> 1] ID 4098:222
2013-08-29 02:47:02.432: [ AGFW][3923724608] {1:3721:2} Preparing START command for: ora.gsd <hostname> 1
2013-08-29 02:47:02.432: [ AGFW][3923724608] {1:3721:2} ora.gsd <hostname> 1 state changed from: UNKNOWN to: STARTING
2013-08-29 02:47:02.433: [ora.gsd][69208384] {1:3721:2} [start] (:CLSN00107:) clsn_agent::start {
2013-08-29 02:47:02.433: [ora.gsd][69208384] {1:3721:2} [start] Arg Value = start
2013-08-29 02:47:02.433: [ora.gsd][69208384] {1:3721:2} [start] getOracleHomeAttrib: oracle_home = <GI_HOME>
2013-08-29 02:47:02.433: [ora.gsd][69208384] {1:3721:2} [start] Utils::getCrsHome crsHome <GI_HOME>
2013-08-29 02:47:02.433: [ora.gsd][69208384] {1:3721:2} [start] Utils::execCmd 1 USR_ORA_ENV: oracleHome:/opt/oracle/oragrid/12.1.0.1 CrsHome:<GI_HOME>
2013-08-29 02:47:02.433: [ora.gsd][69208384] {1:3721:2} [start] Utils::getCrsHome crsHome <GI_HOME>
2013-08-29 02:47:02.433: [ora.gsd][69208384] {1:3721:2} [start] Adding Environment Variables _ORA_AGENT_ACTION=TRUE
2013-08-29 02:47:02.433: [ora.gsd][69208384] {1:3721:2} [start] Adding Environment Variables __IS_HASD_AGENT=
2013-08-29 02:47:02.434: [ora.gsd][69208384] {1:3721:2} [start] Utils:execCmd action = 1 flags = 6 ohome = (null) cmdname = gsdctl.
2013-08-29 02:47:02.434: [ora.gsd][69208384] {1:3721:2} [start] getOracleHomeAttrib: oracle_home = <GI_HOME>
2013-08-29 02:47:02.434: [ora.ons][77621568] {1:3721:2} [start] onsctl start
2013-08-29 02:47:02.434: [ora.ons][77621568] {1:3721:2} [start] getOracleHomeAttrib: oracle_home = <GI_HOME>
2013-08-29 02:47:02.434: [ora.ons][77621568] {1:3721:2} [start] Utils:execCmd action = 1 flags = 6 ohome = <GI_HOME>/opmn/ cmdname = onsctli.
2013-08-29 02:47:02.434: [ora.gsd][69208384] {1:3721:2} [start] (:CLSN00008:)Utils:execCmd scls_process_spawn() failed 1
2013-08-29 02:47:02.434: [ora.gsd][69208384] {1:3721:2} [start] (:CLSN00008:) category: -2, operation: stat, loc: procspawn5, OS error: 2, other: cmd <GI_HOME>/bin/gsdctl not found
2013-08-29 02:47:02.436: [ AGENT][69208384] {1:3721:2} UserErrorException: Locale is
2013-08-29 02:47:02.437: [ora.gsd][69208384] {1:3721:2} [start] clsnUtils::error Exception type=2 string=
CRS-5013: Agent <GI_HOME>/bin/oraagent.bin" failed to start process <GI_HOME>/bin/gsdctl" for action "start": details at "(:CLSN00008:)" in <GI_HOME>/log/<hostname>/agent/crsd/oraagent_oracle/oraagent_oracle.log"

NAME=ora.gsd
TYPE=ora.gsd.type
STATE=OFFLINE
TARGET=OFFLINE
ACL=owner:oracle:rwx,pgrp:oinstall:r-x,other::r--
ACTION_FAILURE_TEMPLATE=
ACTION_SCRIPT=
AGENT_FILENAME=%CRS_HOME%/bin/oraagent%CRS_EXE_SUFFIX%
ALIAS_NAME=ora.%CRS_CSS_NODENAME%.gsd
AUTO_START=always
CHECK_INTERVAL=60
CREATION_SEED=28
DEFAULT_TEMPLATE=
DEGREE=1
DESCRIPTION=Oracle GSD resource
ENABLED=1
ID=ora.gsd
LOAD=1
LOGGING_LEVEL=1
NLS_LANG=
NOT_RESTARTING_TEMPLATE=
OFFLINE_CHECK_INTERVAL=0
PROFILE_CHANGE_TEMPLATE=
RESTART_ATTEMPTS=5
SCRIPT_TIMEOUT=60
START_DEPENDENCIES=
START_TIMEOUT=0
STATE_CHANGE_TEMPLATE=
STOP_DEPENDENCIES=
STOP_TIMEOUT=0
TYPE_VERSION=1.1
UPTIME_THRESHOLD=1d
USR_ORA_ENV=
VERSION=11.2.0.1.0