12.1.0.2 root script fails to start ora.ctssd if nodes name length are not the same

(Doc ID 1918426.1)

Last updated on FEBRUARY 01, 2017

Applies to:

Oracle Database - Enterprise Edition - Version 12.1.0.2 to 12.1.0.2 [Release 12.1]
Information in this document applies to any platform.

Symptoms

Upgrading from 11.2.0.4 to 12.1.0.2, rootupgrade.sh fails on first node as ora.ctssd won't start:


Screen output:

CRS-4123: Oracle High Availability Services has been started.
2014/08/13 13:03:44 CLSRSC-115: Start of resource 'ora.ctssd' failed

2014/08/13 13:03:44 CLSRSC-117: Failed to start Oracle Clusterware stack

2014/08/13 13:03:44 CLSRSC-245: Failed to start Cluster Time Synchronization Service (CTSS)

Died at /usr/app/grid/product/12.1.0.2/crs/install/crsupgrade.pm line 804.
The command '/usr/app/grid/product/12.1.0.2/perl/bin/perl -I/usr/app/grid/product/12.1.0.2/perl/lib -I/us

The issue also happens while running root.sh during fresh install.


<NEW_GI_HOME>/cfgtoollogs/crsconfig/rootcrs_<node>_<timestamp>.log

2014-08-13 12:48:31: CLSRSC-466: Starting shutdown of the current Oracle Grid Infrastructure stack
....
> CRS-2677: Stop of 'ora.ctssd' on 'racn1' succeeded
....
2014-08-13 13:02:50: Start of resource "ora.cssd" Succeeded
2014-08-13 13:02:50: Configured CRS Home: /usr/app/grid/product/12.1.0.2
2014-08-13 13:02:50: Executing cmd: /usr/app/grid/product/12.1.0.2/bin/crsctl start resource ora.ctssd -init -env USR_ORA_ENV=CTSS_REBOOT=TRUE
2014-08-13 13:02:52: Command output:
> CRS-2672: Attempting to start 'ora.ctssd' on 'racn1'
> CRS-2674: Start of 'ora.ctssd' on 'racn1' failed
> CRS-4000: Command Start failed, or completed with errors.
>End Command output
2014-08-13 13:02:52: Configured CRS Home: /usr/app/grid/product/12.1.0.2
2014-08-13 13:02:52: Executing cmd: /usr/app/grid/product/12.1.0.2/bin/crsctl check resource ora.ctssd -init
....
2014-08-13 13:03:39: Executing cmd: /usr/app/grid/product/12.1.0.2/bin/crsctl status resource ora.ctssd -init
2014-08-13 13:03:39: Checking the status of ora.ctssd
2014-08-13 13:03:44: Start of resource "ora.ctssd" failed
CRS-2672: Attempting to start 'ora.ctssd' on 'racn1'
CRS-2674: Start of 'ora.ctssd' on 'racn1' failed
CRS-4000: Command Start failed, or completed with errors.
....
2014-08-13 13:03:44: Executing cmd: /usr/app/grid/product/12.1.0.2/bin/clsecho -p has -f clsrsc -m 115 "ora.ctssd"
2014-08-13 13:03:44: Command output:
> CLSRSC-115: Start of resource 'ora.ctssd' failed
>End Command output
2014-08-13 13:03:44: CLSRSC-115: Start of resource 'ora.ctssd' failed
....
2014-08-13 13:03:44: Executing cmd: /usr/app/grid/product/12.1.0.2/bin/clsecho -p has -f clsrsc -m 117
2014-08-13 13:03:44: Command output:
> CLSRSC-117: Failed to start Oracle Clusterware stack
>End Command output
2014-08-13 13:03:44: CLSRSC-117: Failed to start Oracle Clusterware stack
....
2014-08-13 13:03:44: Executing cmd: /usr/app/grid/product/12.1.0.2/bin/clsecho -p has -f clsrsc -m 245
2014-08-13 13:03:44: Command output:
> CLSRSC-245: Failed to start Cluster Time Synchronization Service (CTSS)
>End Command output
2014-08-13 13:03:44: CLSRSC-245: Failed to start Cluster Time Synchronization Service (CTSS)
2014-08-13 13:03:44: ###### Begin DIE Stack Trace ######
2014-08-13 13:03:44: Package File Line Calling
2014-08-13 13:03:44: --------------- -------------------- ---- ----------
2014-08-13 13:03:44: 1: main rootcrs.pl 267 crsutils::dietrap
2014-08-13 13:03:44: 2: crsupgrade crsupgrade.pm 804 main::__ANON__
2014-08-13 13:03:44: 3: crsupgrade crsupgrade.pm 721 crsupgrade::upgrade_cluster
2014-08-13 13:03:44: 4: crsupgrade crsupgrade.pm 397 crsupgrade::prepare_to_upgrade_clusterware
2014-08-13 13:03:44: 5: crsupgrade crsupgrade.pm 326 crsupgrade::CRSUpgrade
2014-08-13 13:03:44: 6: main rootcrs.pl 276 crsupgrade::new
2014-08-13 13:03:44: ####### End DIE Stack Trace #######


<ADR_BASE>/crs/<node>/crs/trace/alert.log

2014-08-13 13:02:50.886 [OCTSSD(24005)]CRS-8500: Oracle Clusterware OCTSSD process is starting with operating system process ID 24005
2014-08-13 13:02:51.762 [OCTSSD(24005)]CRS-2403: The Cluster Time Synchronization Service on host racn1 is in observer mode.
2014-08-13 13:02:53.011 [ORAROOTAGENT(24042)]CRS-8500: Oracle Clusterware ORAROOTAGENT process is starting with operating system process ID 24042

 

<ADR_BASE>/crs/<node>/crs/trace/octssd.trc

2014-08-13 13:02:51.909103 : CRSCCL:1679406656: gipcListen() Listening on gipcha://racn1:CTSSGROUP_1
2014-08-13 13:02:51.913744 : CRSCCL:1658332928: CSS Group Registration complete.

2014-08-13 13:02:51.914786 : CRSCCL:1658332928: cclGetMemberData called, current mbrcount=2
2014-08-13 13:02:51.914794 : CRSCCL:1658332928: Current map: 0x23eea28: : 1, 2
2014-08-13 13:02:51.914798 : CRSCCL:1658332928: Previous map: 0x7f1562d7df18:


<ADR_BASE>/crs/<node>/crs/trace/ohasd_orarootagent_root.trc

2014-08-13 13:02:50.739055 : AGFW:1849161472: {0:0:172} Agent received the message: RESOURCE_START[ora.ctssd 1 1] ID 4098:622
....
2014-08-13 13:02:51.761668 :CLSDYNAM:1851262720: [ora.ctssd]{0:0:172} [start] PID 24005 from /usr/app/grid/product/12.1.0.2/ctss/init/racn1.pid
2014-08-13 13:02:51.761678 :CLSDYNAM:1851262720: [ora.ctssd]{0:0:172} [start] }DaemonAgent::start exit
2014-08-13 13:02:51.761688 :CLSDYNAM:1851262720: [ora.ctssd]{0:0:172} [start] translateReturnCodes, return = 0, state detail = Checkcb data [0x7f3650053400]: mode[0x40] offset[0 ms].
2014-08-13 13:02:51.761695 :CLSDYNAM:1851262720: [ora.ctssd]{0:0:172} [start] CTSS is in partial
2014-08-13 13:02:52.762421 :GIPCXCPT:1851262720: gipcInternalSend: connection not valid for send operation endp 0x7f3650051190 [00000000000000ac] { gipcEndpoint : localAddr 'ipc', remoteAddr 'ipc://racn1_DBG_CTSSD', numPend 0, numReady 0, numDone 0, numDead 0, numTransfer 0, objFlags 0x0, pidPeer 24005, readyRef (nil), ready 0, wobj 0x7f365005d7c0, sendp 0x7f365005d580 status 0flags 0x2000a61e, flags-2 0x1, usrFlags 0x20020 }, ret gipcretConnectionLost (12)
2014-08-13 13:02:52.762604 :GIPCXCPT:1851262720: gipcSendF [clsdmc_send : clsdmc.c : 728]: EXCEPTION[ ret gipcretConnectionLost (12) ] failed to send on endp 0x7f3650051190 [00000000000000ac] { gipcEndpoint : localAddr 'ipc', remoteAddr 'ipc://racn1_DBG_CTSSD', numPend 0, numReady 0, numDone 0, numDead 0, numTransfer 0, objFlags 0x0, pidPeer 24005, readyRef (nil), ready 0, wobj 0x7f365005d7c0, sendp 0x7f365005d580 status 0flags 0x2000a61e, flags-2 0x1, usrFlags 0x20020 }, addr 0000000000000000, buf 0x7f36500629e0, len 65, cookie (nil), flags 0x0
CLSDMC:1851262720: Failed to send dynamic control message to connection [ipc://racn1_DBG_CTSSD][12]
2014-08-13 13:02:52.762693 : CLSDMC:1851262720: gipcWait gets wrong msg from connection [ipc://racn1_DBG_CTSSD][0] with type gipcreqtypeDisconnect
2014-08-13 13:02:52.763402 :CLSDYNAM:1851262720: [ora.ctssd]{0:0:172} [start] ClsdmClient::sendMessage clsdmc_send error rmsg:0 ecode:-10 errbuf:CRS-02004: error 0 encountered when sending messages to CTSSD
....
2014-08-13 13:02:52.771438 : AGFW:1849161472: {0:0:172} ora.ctssd 1 1 state changed from: STARTING to: OFFLINE
2014-08-13 13:02:52.771697 : AGFW:1849161472: {0:0:172} Agent sending last reply for: RESOURCE_START[ora.ctssd 1 1] ID 4098:622
2014-08-13 13:02:52.780011 : AGFW:1851262720: {0:0:172} Agent has no resources to be monitored, Shutting down ..
2014-08-13 13:02:52.780074 : AGFW:1851262720: {0:0:172} Agent sending message to PE: AGENT_SHUTDOWN_REQUEST[Proxy] ID 20486:31
2014-08-13 13:02:52.782184 : AGFW:1849161472: {0:0:172} Agent is shutting down.
2014-08-13 13:02:52.782207 : AGENT:1849161472: {0:0:172} Agfw calling user exitCB, will exit on return
2014-08-13 13:02:52.782222 : AGENT:1849161472: {0:0:172} returned from user exitCB, exiting
2014-08-13 13:02:52.782273 : AGFW:1849161472: {0:0:172} Agent is exiting with exit code: 1

 

 

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms