GI 12c/12.1.0.x rootupgrade.sh fails: PRCR-1065 : Failed to stop resource ora.gsd

(Doc ID 1580360.1)

Last updated on APRIL 17, 2016

Applies to:

Oracle Database - Enterprise Edition - Version 12.1.0.1 and later
Information in this document applies to any platform.

Symptoms

Upgrading to GI 12.1.0.1, rootupgrade.sh failed on first node

 

Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
PRCR-1065 : Failed to stop resource ora.gsd
CRS-5013: Agent "/opt/oracle/oragrid/12.1.0.1/bin/oraagent.bin" failed to start process "/opt/oracle/oragrid/12.1.0.1/bin/gsdctl" for action "clean": details at "(:CLSN00008:)" in "/opt/oracle/oragrid/12.1.0.1/log/usfldbh2/agent/crsd/oraagent_oracle/oraagent_oracle.log"
CRS-5013: Agent "/opt/oracle/oragrid/12.1.0.1/bin/oraagent.bin" failed to start process "/opt/oracle/oragrid/12.1.0.1/bin/gsdctl" for action "check": details at "(:CLSN00008:)" in "/opt/oracle/oragrid/12.1.0.1/log/usfldbh2/agent/crsd/oraagent_oracle/oraagent_oracle.log"
CRS-2680: Clean of 'ora.gsd' on 'usfldbh2' failed
2013/08/29 02:48:20 CLSRSC-180: An error occurred while executing the command 'srvctl upgrade model -s 11.2.0.3.0 -d 12.1.0.1.0  -p first' (error code 0)
2013/08/29 02:48:20 CLSRSC-280: Failed to perform first node tasks for cluster modeling upgrade
Died at /opt/oracle/oragrid/12.1.0.1/crs/install/crsupgrade.pm line 3254.
The command '/opt/oracle/oragrid/12.1.0.1/perl/bin/perl -I/opt/oracle/oragrid/12.1.0.1/perl/lib -I/opt/oracle/oragrid/12.1.0.1/crs/install /opt/oracle/oragrid/12.1.0.1/crs/install/rootcrs.pl  -upgrade -auto' execution failed

 

2013-08-29 02:47:22: The software version on usfldbh2 is 12.1.0.1.0
2013-08-29 02:47:22: Invoking "/opt/oracle/oragrid/12.1.0.1/bin/srvctl upgrade model  -s 11.2.0.3.0 -d 12.1.0.1.0 -p first"
2013-08-29 02:47:22: trace file=/opt/oracle/oragrid/12.1.0.1/cfgtoollogs/crsconfig/srvmcfg0.log
2013-08-29 02:47:22: Executing /opt/oracle/oragrid/12.1.0.1/bin/srvctl upgrade model  -s 11.2.0.3.0 -d 12.1.0.1.0 -p first
2013-08-29 02:47:22: Executing cmd: /opt/oracle/oragrid/12.1.0.1/bin/srvctl upgrade model  -s 11.2.0.3.0 -d 12.1.0.1.0 -p first
2013-08-29 02:48:20: Command output:
>  PRCR-1065 : Failed to stop resource ora.gsd
>  CRS-5013: Agent "/opt/oracle/oragrid/12.1.0.1/bin/oraagent.bin" failed to start process "/opt/oracle/oragrid/12.1.0.1/bin/gsdctl" for action "clean": details at "(:CLSN00008:)" in "/opt/oracle/oragrid/12.1.0.1/log/usfldbh2/agent/crsd/oraagent_oracle/oraagent_oracle.log"
>  CRS-5013: Agent "/opt/oracle/oragrid/12.1.0.1/bin/oraagent.bin" failed to start process "/opt/oracle/oragrid/12.1.0.1/bin/gsdctl" for action "check": details at "(:CLSN00008:)" in "/opt/oracle/oragrid/12.1.0.1/log/usfldbh2/agent/crsd/oraagent_oracle/oraagent_oracle.log"
>  CRS-2680: Clean of 'ora.gsd' on 'usfldbh2' failed
>End Command output
2013-08-29 02:48:20:   "/opt/oracle/oragrid/12.1.0.1/bin/srvctl upgrade model  -s 11.2.0.3.0 -d 12.1.0.1.0 -p first" failed with status 1.
2013-08-29 02:48:20: Executing cmd: /opt/oracle/oragrid/12.1.0.1/bin/clsecho -p has -f clsrsc -m 180 "srvctl upgrade model -s 11.2.0.3.0 -d 12.1.0.1.0  -p first" "0"
2013-08-29 02:48:20: Command output:
>  CLSRSC-180: An error occurred while executing the command 'srvctl upgrade model -s 11.2.0.3.0 -d 12.1.0.1.0  -p first' (error code 0)
>End Command output
2013-08-29 02:48:20: Executing cmd: /opt/oracle/oragrid/12.1.0.1/bin/clsecho -p has -f clsrsc -m 180 "srvctl upgrade model -s 11.2.0.3.0 -d 12.1.0.1.0  -p first" "0"
2013-08-29 02:48:20: Command output:
>  CLSRSC-180: An error occurred while executing the command 'srvctl upgrade model -s 11.2.0.3.0 -d 12.1.0.1.0  -p first' (error code 0)
>End Command output
2013-08-29 02:48:20: CLSRSC-180: An error occurred while executing the command 'srvctl upgrade model -s 11.2.0.3.0 -d 12.1.0.1.0  -p first' (error code 0)
2013-08-29 02:48:20: Running as user oracle: /opt/oracle/oragrid/12.1.0.1/bin/cluutil -ckpt -oraclebase /opt/oracle/orabase -writeckpt -name ROOTCRS_NODECONFIG -state FAIL
2013-08-29 02:48:20: s_run_as_user2: Running /bin/su oracle -c ' /opt/oracle/oragrid/12.1.0.1/bin/cluutil -ckpt -oraclebase /opt/oracle/orabase -writeckpt -name ROOTCRS_NODECONFIG -state FAIL '
2013-08-29 02:48:20: Removing file /tmp/fileVzyfMf
2013-08-29 02:48:20: Successfully removed file: /tmp/fileVzyfMf
2013-08-29 02:48:20: /bin/su successfully executed

2013-08-29 02:48:20: Succeeded in writing the checkpoint:'ROOTCRS_NODECONFIG' with status:FAIL
2013-08-29 02:48:20: Executing cmd: /opt/oracle/oragrid/12.1.0.1/bin/clsecho -p has -f clsrsc -m 280
2013-08-29 02:48:20: Command output:
>  CLSRSC-280: Failed to perform first node tasks for cluster modeling upgrade
>End Command output
2013-08-29 02:48:20: Executing cmd: /opt/oracle/oragrid/12.1.0.1/bin/clsecho -p has -f clsrsc -m 280
2013-08-29 02:48:20: Command output:
>  CLSRSC-280: Failed to perform first node tasks for cluster modeling upgrade
>End Command output
2013-08-29 02:48:20: CLSRSC-280: Failed to perform first node tasks for cluster modeling upgrade
2013-08-29 02:48:20: ###### Begin DIE Stack Trace ######
2013-08-29 02:48:20:     Package         File                 Line Calling  
2013-08-29 02:48:20:     --------------- -------------------- ---- ----------
2013-08-29 02:48:20:  1: main            rootcrs.pl            211 crsutils::dietrap
2013-08-29 02:48:20:  2: crsupgrade      crsupgrade.pm        3254 main::__ANON__
2013-08-29 02:48:20:  3: crsupgrade      crsupgrade.pm         239 crsupgrade::upgrade_node
2013-08-29 02:48:20:  4: crsupgrade      crsupgrade.pm         190 crsupgrade::CRSUpgrade
2013-08-29 02:48:20:  5: main            rootcrs.pl            220 crsupgrade::new
2013-08-29 02:48:20: ####### End DIE Stack Trace #######

2013-08-29 02:48:20: ROOTCRS_NODECONFIG checkpoint has failed
2013-08-29 02:48:20: Running as user oracle: /opt/oracle/oragrid/12.1.0.1/bin/cluutil -ckpt -oraclebase /opt/oracle/orabase -writeckpt -name ROOTCRS_NODECONFIG -state FAIL
2013-08-29 02:48:20: s_run_as_user2: Running /bin/su oracle -c ' /opt/oracle/oragrid/12.1.0.1/bin/cluutil -ckpt -oraclebase /opt/oracle/orabase -writeckpt -name ROOTCRS_NODECONFIG -state FAIL '
2013-08-29 02:48:20: Removing file /tmp/filegNLxAd
2013-08-29 02:48:20: Successfully removed file: /tmp/filegNLxAd
2013-08-29 02:48:20: /bin/su successfully executed

 

2013-08-29 02:46:56.309: [ora.gsd][71309632] {1:3721:2} [check] Utils:execCmd action = 3 flags = 6 ohome = (null) cmdname = gsdctl.
2013-08-29 02:46:56.310: [ora.gsd][71309632] {1:3721:2} [check] getOracleHomeAttrib: oracle_home = /opt/oracle/oragrid/12.1.0.1
2013-08-29 02:46:56.312: [ora.gsd][71309632] {1:3721:2} [check] (:CLSN00008:)Utils:execCmd scls_process_spawn() failed 1
2013-08-29 02:46:56.312: [ora.gsd][71309632] {1:3721:2} [check] (:CLSN00008:) category: -2, operation: stat, loc: procspawn5, OS error: 2, other: cmd /opt/oracle/oragrid/12.1.0.1/bin/gsdctl not found
2013-08-29 02:46:56.313: [ora.gsd][71309632] {1:3721:2} [check] clsnUtils::error Exception type=2 string=
CRS-5013: Agent "/opt/oracle/oragrid/12.1.0.1/bin/oraagent.bin" failed to start process "/opt/oracle/oragrid/12.1.0.1/bin/gsdctl" for action "check": details at "(:CLSN00008:)" in "/opt/oracle/oragrid/12.1.0.1/log/usfldbh2/agent/crsd/oraagent_oracle/oraagent_oracle.log"


2013-08-29 02:47:02.432: [    AGFW][3923724608] {1:3721:2} Agent received the message: RESOURCE_START[ora.gsd usfldbh2 1] ID 4098:222
2013-08-29 02:47:02.432: [    AGFW][3923724608] {1:3721:2} Preparing START command for: ora.gsd usfldbh2 1
2013-08-29 02:47:02.432: [    AGFW][3923724608] {1:3721:2} ora.gsd usfldbh2 1 state changed from: UNKNOWN to: STARTING
2013-08-29 02:47:02.433: [ora.gsd][69208384] {1:3721:2} [start] (:CLSN00107:) clsn_agent::start {
2013-08-29 02:47:02.433: [ora.gsd][69208384] {1:3721:2} [start] Arg Value = start
2013-08-29 02:47:02.433: [ora.gsd][69208384] {1:3721:2} [start] getOracleHomeAttrib: oracle_home = /opt/oracle/oragrid/12.1.0.1
2013-08-29 02:47:02.433: [ora.gsd][69208384] {1:3721:2} [start] Utils::getCrsHome crsHome /opt/oracle/oragrid/12.1.0.1
2013-08-29 02:47:02.433: [ora.gsd][69208384] {1:3721:2} [start] Utils::execCmd 1 USR_ORA_ENV: oracleHome:/opt/oracle/oragrid/12.1.0.1 CrsHome:/opt/oracle/oragrid/12.1.0.1
2013-08-29 02:47:02.433: [ora.gsd][69208384] {1:3721:2} [start] Utils::getCrsHome crsHome /opt/oracle/oragrid/12.1.0.1
2013-08-29 02:47:02.433: [ora.gsd][69208384] {1:3721:2} [start] Adding Environment Variables _ORA_AGENT_ACTION=TRUE
2013-08-29 02:47:02.433: [ora.gsd][69208384] {1:3721:2} [start] Adding Environment Variables __IS_HASD_AGENT=
2013-08-29 02:47:02.434: [ora.gsd][69208384] {1:3721:2} [start] Utils:execCmd action = 1 flags = 6 ohome = (null) cmdname = gsdctl.
2013-08-29 02:47:02.434: [ora.gsd][69208384] {1:3721:2} [start] getOracleHomeAttrib: oracle_home = /opt/oracle/oragrid/12.1.0.1
2013-08-29 02:47:02.434: [ora.ons][77621568] {1:3721:2} [start] onsctl start
2013-08-29 02:47:02.434: [ora.ons][77621568] {1:3721:2} [start] getOracleHomeAttrib: oracle_home = /opt/oracle/oragrid/12.1.0.1
2013-08-29 02:47:02.434: [ora.ons][77621568] {1:3721:2} [start] Utils:execCmd action = 1 flags = 6 ohome = /opt/oracle/oragrid/12.1.0.1/opmn/ cmdname = onsctli.
2013-08-29 02:47:02.434: [ora.gsd][69208384] {1:3721:2} [start] (:CLSN00008:)Utils:execCmd scls_process_spawn() failed 1
2013-08-29 02:47:02.434: [ora.gsd][69208384] {1:3721:2} [start] (:CLSN00008:) category: -2, operation: stat, loc: procspawn5, OS error: 2, other: cmd /opt/oracle/oragrid/12.1.0.1/bin/gsdctl not found
2013-08-29 02:47:02.436: [   AGENT][69208384] {1:3721:2} UserErrorException: Locale is
2013-08-29 02:47:02.437: [ora.gsd][69208384] {1:3721:2} [start] clsnUtils::error Exception type=2 string=
CRS-5013: Agent "/opt/oracle/oragrid/12.1.0.1/bin/oraagent.bin" failed to start process "/opt/oracle/oragrid/12.1.0.1/bin/gsdctl" for action "start": details at "(:CLSN00008:)" in "/opt/oracle/oragrid/12.1.0.1/log/usfldbh2/agent/crsd/oraagent_oracle/oraagent_oracle.log"

 

NAME=ora.gsd
TYPE=ora.gsd.type
STATE=OFFLINE
TARGET=OFFLINE
ACL=owner:oracle:rwx,pgrp:oinstall:r-x,other::r--
ACTION_FAILURE_TEMPLATE=
ACTION_SCRIPT=
AGENT_FILENAME=%CRS_HOME%/bin/oraagent%CRS_EXE_SUFFIX%
ALIAS_NAME=ora.%CRS_CSS_NODENAME%.gsd
AUTO_START=always
CHECK_INTERVAL=60
CREATION_SEED=28
DEFAULT_TEMPLATE=
DEGREE=1
DESCRIPTION=Oracle GSD resource
ENABLED=1
ID=ora.gsd
LOAD=1
LOGGING_LEVEL=1
NLS_LANG=
NOT_RESTARTING_TEMPLATE=
OFFLINE_CHECK_INTERVAL=0
PROFILE_CHANGE_TEMPLATE=
RESTART_ATTEMPTS=5
SCRIPT_TIMEOUT=60
START_DEPENDENCIES=
START_TIMEOUT=0
STATE_CHANGE_TEMPLATE=
STOP_DEPENDENCIES=
STOP_TIMEOUT=0
TYPE_VERSION=1.1
UPTIME_THRESHOLD=1d
USR_ORA_ENV=
VERSION=11.2.0.1.0

 

 

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms