Clusterware do not start on ALL nodes after reboot
(Doc ID 1676719.1)
Last updated on DECEMBER 03, 2024
Applies to:
Oracle Database - Enterprise Edition - Version 11.2.0.1 to 11.2.0.4 [Release 11.2]Oracle Database Cloud Schema Service - Version N/A and later
Oracle Database Exadata Cloud Machine - Version N/A and later
Oracle Database Exadata Express Cloud Service - Version N/A and later
Oracle Cloud Infrastructure - Database Service - Version N/A and later
Information in this document applies to any platform.
Symptoms
Clusterware fails to start on ALL nodes after the cluster nodes are rebooted. Nodes in cluster are mdbp01,mdbp02,mdbp03 and mdbp04 with cluster name as "crs".
$GRID_HOME/log/<node>/gpnpd/gpnpd.log report below errors:
2014-01-23 01:56:50.376: [ GPNP][7]clsgpnpd_validateProfile: [at clsgpnpd.c:2888] Result: (89) CLSGPNP_SIG_WALLETDIF. Profile failed to verify. prf=6000000000d99e30
2014-01-23 01:56:50.376: [ GPNP][7]clsgpnpd_putProfileDo: [at clsgpnpd.c:5336] Result: (89) CLSGPNP_SIG_WALLETDIF. PUT>>> REFUSED best p=6000000000d99e30 from "tcp://rdbp08:10653"
2014-01-23 01:56:50.378: [ GPNP][7]clsgpnp_profileCallUrlInt: [at clsgpnp.c:2104] put-profile call to url "tcp://rdbp06:50099" disco "mdns:service:gpnp._tcp.local.://rdbp06:50099/agent=gpnpd,cname=crs,host=rdbp06,pid=6049/gpnpd h:rdbp06 c:crs" [f=0 claimed- host:mdbp01 cname:crs seq:6 auth:CN=GPnP_peer]
2014-01-23 01:56:52.111: [ OCRMSG][3]GIPC error [29] msg [gipcretConnectionRefused]
2014-01-23 01:57:06.251: [ OCRMSG][3]GIPC error [29] msg [gipcretConnectionRefused]
2014-01-23 01:57:27.462: [ OCRMSG][3]GIPC error [29] msg [gipcretConnectionRefused]
$GRID_HOME/log/<node>/cssd/ocssd.log shows:
2014-01-23 02:37:21.387: [ CSSD][5]clssgmEvtInformation: reqtype (11) req (6000000000c081f0)
2014-01-23 02:37:21.387: [ CSSD][5]clssnmQueueNotification: type (11) 6000000000c081f0
2014-01-23 02:37:22.639: [ GPNP][1]clsgpnpm_newWiredMsg: [at clsgpnpm.c:741] Msg-reply has soap fault 10 (Operation returned Retry (error CLSGPNP_CALL_AGAIN)) [uri http://www.grid-pnp.org/2005/12/gpnp-errors#"]
2014-01-23 02:37:24.659: [ GPNP][1]clsgpnpm_newWiredMsg: [at clsgpnpm.c:741] Msg-reply has soap fault 10 (Operation returned Retry (error CLSGPNP_CALL_AGAIN)) [uri http://www.grid-pnp.org/2005/12/gpnp-errors#"]
2014-01-23 02:37:26.679: [ GPNP][1]clsgpnpm_newWiredMsg: [at clsgpnpm.c:741] Msg-reply has soap fault 10 (Operation returned Retry (error CLSGPNP_CALL_AGAIN)) [uri http://www.grid-pnp.org/2005/12/gpnp-errors#"]
...
...
2014-01-23 02:47:18.568: [ GPNP][1]clsgpnpm_newWiredMsg: [at clsgpnpm.c:741] Msg-reply has soap fault 10 (Operation returned Retry (error CLSGPNP_CALL_AGAIN)) [uri http://www.grid-pnp.org/2005/12/gpnp-errors#"]
2014-01-23 02:47:19.377: [ CSSD][5]clssgmExecuteClientRequest: MAINT recvd from proc 3 (6000000000c1bb00)
2014-01-23 02:47:19.377: [ CSSD][5]clssgmShutDown: Received abortive shutdown request from client.
2014-01-23 02:47:19.377: [ CSSD][5]###################################
2014-01-23 02:47:19.377: [ CSSD][5]clssscExit: CSSD aborting from thread GMClientListener
2014-01-23 02:47:19.377: [ CSSD][5]###################################
2014-01-23 02:47:19.377: [ CSSD][5](:CSSSC00012:)clssscExit: A fatal error occurred and the CSS daemon is terminating abnormally
Cause
To view full details, sign in with your My Oracle Support account. |
|
Don't have a My Oracle Support account? Click to get started! |
In this Document
| Symptoms |
| Cause |
| Solution |