AIX: root.sh fails on non-first(second or other) node: CLSRSC-117: Failed to start Oracle Clusterware stack

(Doc ID 2216449.1)

Last updated on DECEMBER 25, 2016

Applies to:

Oracle Database - Enterprise Edition - Version 12.1.0.2 and later
IBM AIX on POWER Systems (64-bit)

Symptoms

Installing Grid Infrastructure on AIX, root.sh succeeded on node1 but fails on node2:

2016-11-03 15:43:22: Executing cmd: /u02/grid/12.1.0/bin/crsctl start crs -wait
2016-11-03 15:54:01: Command output:
> CRS-4123: Starting Oracle High Availability Services-managed resources
> CRS-2672: Attempting to start 'ora.mdnsd' on 'aixora2racnode01'
> CRS-2672: Attempting to start 'ora.evmd' on 'aixora2racnode01'
> CRS-2676: Start of 'ora.mdnsd' on 'aixora2racnode01' succeeded
> CRS-2676: Start of 'ora.evmd' on 'aixora2racnode01' succeeded
....
> CRS-2672: Attempting to start 'ora.cssd' on 'aixora2racnode01'
....
> CRS-2883: Resource 'ora.cssd' failed during Clusterware stack start.
> CRS-4406: Oracle High Availability Services synchronous start failed.
> CRS-4000: Command Start failed, or completed with errors.
>End Command output
2016-11-03 15:54:01: The return value of blocking start of CRS: 1
2016-11-03 15:54:01: Failed to start the Oracle CRS stack
2016-11-03 15:54:01: Executing cmd: /u02/grid/12.1.0/bin/clsecho -p has -f clsrsc -m 117
2016-11-03 15:54:03: Command output:
> CLSRSC-117: Failed to start Oracle Clusterware stack
>End Command output

 

cssd agent trace 

2016-11-03 15:46:18.656627 : USRTHRD:515: wait type 20 rc 0
2016-11-03 15:46:18.656647 : USRTHRD:515: clsnpoll_cssini: node number: 2
2016-11-03 15:46:18.656683 : USRTHRD:515: clsnpoll_WaitCSS: waiting for first reconfiguration and kgzf initialization
2016-11-03 15:46:19.666479 : USRTHRD:515: clsnpoll_GetValue: get type 18 ret 16 req 0
2016-11-03 15:46:20.666765 : USRTHRD:515: clsnpoll_GetValue: get type 18 ret 16 req 0
2016-11-03 15:46:21.674212 : USRTHRD:515: clsnpoll_GetValue: get type 18 ret 16 req 0
2016-11-03 15:46:22.687632 : USRTHRD:515: clsnpoll_GetValue: get type 18 ret 16 req 0
2016-11-03 15:46:23.695417 : USRTHRD:515: clsnpoll_GetValue: get type 18 ret 16 req 0
2016-11-03 15:46:24.696478 : USRTHRD:515: clsnpoll_GetValue: get type 18 ret 16 req 0
....
2016-11-03 15:53:47.415849 : USRTHRD:515: clsnpoll_GetValue: get type 18 ret 16 req 0
2016-11-03 15:53:48.416777 : USRTHRD:515: clsnpoll_GetValue: get type 18 ret 16 req 0
2016-11-03 15:53:48.505093 : AGENT:1800: {0:9:3} {0:9:3} Created alert : (:CRSAGF00113:) : Aborting the command: start for resource: ora.cssd 1 1
2016-11-03 15:53:48.505167 :CLSDYNAM:1800: [ora.cssd]{0:9:3} [start] clsncssd_cssdabort: sending shutdown abort to CSS with new ctx
2016-11-03 15:53:48.525136 : USRTHRD:515: clsnpoll_GetValue: get type 18 ret 0 req 0
2016-11-03 15:53:48.525168 : USRTHRD:515: clsnpoll_GetValue: error in gipcWait 0
2016-11-03 15:53:48.525193 : USRTHRD:515: clsnpoll_WaitCSS: completed with status 3
2016-11-03 15:53:48.525227 : USRTHRD:515: clsnpoll_initialize: problems in the first reconfiguration of CSS - 3
2016-11-03 15:53:48.525250 : USRTHRD:515: clsnpoll_cleanup: to exit status = 3
2016-11-03 15:53:48.525386 : USRTHRD:515: clsnpoll_cleanup: failure, sending shutdown immediate to CSS
2016-11-03 15:53:48.525421 : USRTHRD:515: clsnpoll_cleanup: CSS is down
2016-11-03 15:53:48.535391 : USRTHRD:2829: clsnproc_main1: in the main loop status=0 stop=0 fhb=0 gone=1 shut=0
2016-11-03 15:53:48.586499 :CLSDYNAM:2057: [ora.cssd]{0:9:3} [start] clsncssd_waitpoll: signal abort thread.
2016-11-03 15:53:48.586542 :CLSDYNAM:2057: [ora.cssd]{0:9:3} [start] clsncssd_waitpoll: agent is not up.
2016-11-03 15:53:58.606608 :CLSDYNAM:1800: [ora.cssd]{0:9:3} [start] clsncssd_abortaction: abort action done
2016-11-03 15:53:58.606709 : AGFW:1800: {0:9:3} Command: start for resource: ora.cssd 1 1 completed with status: TIMEDOUT
2016-11-03 15:53:58.613062 : USRTHRD:1800: {0:9:3} target_mode: hub 1 config_mode auto 0
2016-11-03 15:53:58.613133 :CLSDYNAM:1800: [ora.cssd]{0:9:3} [check] clsncssd_getattr: attr RIM_TERMWAIT, value 20
2016-11-03 15:53:58.613159 : USRTHRD:1800: {0:9:3} last rim seconds 20000
2016-11-03 15:53:58.613209 :CLSDYNAM:1800: [ora.cssd]{0:9:3} [check] clsncssd_getattr: attr LOGGING_LEVEL, value 1

 

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms