Clusterware do not start on ALL nodes after reboot (Doc ID 1676719.1)

Last updated on OCTOBER 13, 2016

Applies to:

Oracle Database - Enterprise Edition - Version 11.2.0.1 to 11.2.0.4 [Release 11.2]
Information in this document applies to any platform.

Symptoms

Clusterware fails to start on ALL nodes after the cluster nodes are rebooted.  Nodes in cluster are mdbp01,mdbp02,mdbp03 and mdbp04 with cluster name as "crs".

$GRID_HOME/log/<node>/gpnpd/gpnpd.log report below errors:

2014-01-23 01:56:50.376: [ CLSXSEC][7]clsxsec_CtxCKVerify: [at clsxsec.c:1768] Result: (10017) CLSXERR_SEC_BSAFE_DATA. Failed to verify: nzerr=29237 vstat=2             
2014-01-23 01:56:50.376: [    GPNP][7]clsgpnpd_validateProfile: [at clsgpnpd.c:2888] Result: (89) CLSGPNP_SIG_WALLETDIF. Profile failed to verify.  prf=6000000000d99e30
2014-01-23 01:56:50.376: [    GPNP][7]clsgpnpd_putProfileDo: [at clsgpnpd.c:5336] Result: (89) CLSGPNP_SIG_WALLETDIF. PUT>>> REFUSED best p=6000000000d99e30 from "tcp://rdbp08:10653"
2014-01-23 01:56:50.378: [    GPNP][7]clsgpnp_profileCallUrlInt: [at clsgpnp.c:2104] put-profile call to url "tcp://rdbp06:50099" disco "mdns:service:gpnp._tcp.local.://rdbp06:50099/agent=gpnpd,cname=crs,host=rdbp06,pid=6049/gpnpd h:rdbp06 c:crs" [f=0 claimed- host:mdbp01 cname:crs seq:6 auth:CN=GPnP_peer] 
2014-01-23 01:56:52.111: [  OCRMSG][3]GIPC error [29] msg [gipcretConnectionRefused]
2014-01-23 01:57:06.251: [  OCRMSG][3]GIPC error [29] msg [gipcretConnectionRefused]
2014-01-23 01:57:27.462: [  OCRMSG][3]GIPC error [29] msg [gipcretConnectionRefused]

 $GRID_HOME/log/<node>/cssd/ocssd.log shows:

2014-01-23 02:37:21.387: [    CSSD][5]clssgmEvtInformation: reqtype (11) req (6000000000c081f0)
2014-01-23 02:37:21.387: [    CSSD][5]clssnmQueueNotification: type (11) 6000000000c081f0
2014-01-23 02:37:22.639: [    GPNP][1]clsgpnpm_newWiredMsg: [at clsgpnpm.c:741] Msg-reply has soap fault 10 (Operation returned Retry (error CLSGPNP_CALL_AGAIN)) [uri http://www.grid-pnp.org/2005/12/gpnp-errors#"]
2014-01-23 02:37:24.659: [    GPNP][1]clsgpnpm_newWiredMsg: [at clsgpnpm.c:741] Msg-reply has soap fault 10 (Operation returned Retry (error CLSGPNP_CALL_AGAIN)) [uri http://www.grid-pnp.org/2005/12/gpnp-errors#"]
2014-01-23 02:37:26.679: [    GPNP][1]clsgpnpm_newWiredMsg: [at clsgpnpm.c:741] Msg-reply has soap fault 10 (Operation returned Retry (error CLSGPNP_CALL_AGAIN)) [uri http://www.grid-pnp.org/2005/12/gpnp-errors#"]
...
...
2014-01-23 02:47:18.568: [    GPNP][1]clsgpnpm_newWiredMsg: [at clsgpnpm.c:741] Msg-reply has soap fault 10 (Operation returned Retry (error CLSGPNP_CALL_AGAIN)) [uri http://www.grid-pnp.org/2005/12/gpnp-errors#"]
2014-01-23 02:47:19.377: [    CSSD][5]clssgmExecuteClientRequest: MAINT recvd from proc 3 (6000000000c1bb00)
2014-01-23 02:47:19.377: [    CSSD][5]clssgmShutDown: Received abortive shutdown request from client.
2014-01-23 02:47:19.377: [    CSSD][5]###################################
2014-01-23 02:47:19.377: [    CSSD][5]clssscExit: CSSD aborting from thread GMClientListener
2014-01-23 02:47:19.377: [    CSSD][5]###################################
2014-01-23 02:47:19.377: [    CSSD][5](:CSSSC00012:)clssscExit: A fatal error occurred and the CSS daemon is terminating abnormally

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms