CSSD Fails to Start After Repeated Message: Msg-reply has soap fault 10 (Operation returned Retry (error CLSGPNP_CALL_AGAIN)) (Doc ID 1588034.1)

2013-09-08 10:14:02.958: [ GPNP][1]clsgpnpm_newWiredMsg: [at clsgpnpm.c:741] Msg-reply has soap fault 10 (Operation returned Retry (error CLSGPNP_CALL_AGAIN)) [uri "http://www.grid-pnp.org/2005/12/gpnp-errors#"]
2013-09-08 10:14:04.967: [ GPNP][1]clsgpnpm_newWiredMsg: [at clsgpnpm.c:741] Msg-reply has soap fault 10 (Operation returned Retry (error CLSGPNP_CALL_AGAIN)) [uri "http://www.grid-pnp.org/2005/12/gpnp-errors#"]
....
....  ====>>>> after about 10 minutes
....
2013-09-08 10:23:57.814: [ GPNP][1]clsgpnpm_newWiredMsg: [at clsgpnpm.c:741] Msg-reply has soap fault 10 (Operation returned Retry (error CLSGPNP_CALL_AGAIN)) [uri "http://www.grid-pnp.org/2005/12/gpnp-errors#"]
2013-09-08 10:23:59.708: [ CSSD][5]clssgmExecuteClientRequest: MAINT recvd from proc 2 (6000000000c704b0)
2013-09-08 10:23:59.708: [ CSSD][5]clssgmShutDown: Received abortive shutdown request from client.
2013-09-08 10:23:59.708: [ CSSD][5]###################################
2013-09-08 10:23:59.708: [ CSSD][5]clssscExit: CSSD aborting from thread GMClientListener
2013-09-08 10:23:59.708: [ CSSD][5]###################################
2013-09-08 10:23:59.708: [ CSSD][5](:CSSSC00012:)clssscExit: A fatal error occurred and the CSS daemon is terminating abnormally

2013-09-08 10:14:03.987: [ GPNP][7]clsgpnp_profileCallUrlInt: [at clsgpnp.c:2104] put-profile call to url "tcp://rac08:59327" disco "mdns:service:gpnp._tcp.local.://rac08:59327/agent=gpnpd,cname=ngprod_cluster,host=<HOSTNAME1>,pid=6271/gpnpd h:rac08 c:ngprod_cluster" [f=0 claimed- host:rac11 cname:ngprod_cluster seq:5 auth:CN=GPnP_peer]
2013-09-08 10:14:04.751: [ OCRMSG][3]GIPC error [29] msg [gipcretConnectionRefused]
2013-09-08 10:24:40.078: [ OCRMSG][3]GIPC error [29] msg [gipcretConnectionRefused]
2013-09-08 10:24:47.680: [ GPNP][7]clsgpnp_profileCallUrlInt: [at clsgpnp.c:2234] Result: (25) CLSGPNP_DUPLICATE. Successful put-profile CALL to remote "tcp://rac08:59327" disco "mdns:service:gpnp._tcp.local.://rac08:59327/agent=gpnpd,cname=ngprod_cluster,host=<HOSTNAME1>,pid=6271/gpnpd h:rac08 c:ngprod_cluster"
2013-09-08 10:25:52.547: [ GPNP][7]clsgpnp_profileCallUrlInt: [at clsgpnp.c:2104] put-profile call to url "tcp://rac10:49227" disco "mdns:service:gpnp._tcp.local.://rac10:49227/agent=gpnpd,cname=ngprod_cluster,host=<HOSTNAME1>,pid=3861/gpnpd h:rac10 c:ngprod_cluster" [f=0 claimed- host:rac11 cname:ngprod_cluster seq:5 auth:CN=GPnP_peer]
2013-09-08 10:26:10.984: [ OCRMSG][3]GIPC error [29] msg [gipcretConnectionRefused]
2013-09-08 10:33:45.511: [ OCRMSG][3]GIPC error [29] msg [gipcretConnectionRefused]
2013-09-08 10:34:12.103: [ GPNP][4]CLSDM requested exit
2013-09-08 10:34:12.104: [ default][4]GPNPD on node corpng11 shutdown.
2013-09-08 10:48:05.861: [ default][1]

 

====>>>> The put profile takes very long time, the following never appeared in gpnpd.log

[ GPNP][7]clsgpnpd_pushThread: [at clsgpnpd.c:4770] START gpnpd start serving clients after profile updates

2013-09-08 10:23:59.706
[/ora00/app/11203/grid/bin/cssdagent(4818)]CRS-5818:Aborted command 'start' for resource 'ora.cssd'. Details at (:CRSAGF00113:) {0:0:2} in /ora00/app/11203/grid/log/rac11/agent/ohasd/oracssdagent_root/oracssdagent_root.log.
2013-09-08 10:23:59.708
[cssd(4824)]CRS-1656:The CSS daemon is terminating due to a fatal error; Details at (:CSSSC00012:) in /ora00/app/11203/grid/log/rac11/cssd/ocssd.log
2013-09-08 10:23:59.709
[cssd(4824)]CRS-1603:CSSD on node rac11 shutdown by user.

2013-09-08 10:37:56.671: [ OCRMSG][6]prom_recv: Failed to receieve [12]
2013-09-08 10:37:56.672: [ OCRMSG][6]GIPC error [12] msg [gipcretConnectionLost]
2013-09-08 10:37:56.672: [ OCRAPI][6]procr_open_key_ext: Fail Fast flag is set and OCR server is not found. Exiting the API.
2013-09-08 10:37:56.672: [ GPNP][6]procr_open_key_ext: OCR api procr_open_key_ext failed for key SYSTEM.GPnP.profiles.peer.best
2013-09-08 10:37:56.672: [ GPNP][6]procr_open_key_ext: OCR current boot level : 7
2013-09-08 10:37:56.672: [ GPNP][6]procr_open_key_ext: OCR error code : 32
2013-09-08 10:37:56.672: [ GPNP][6]procr_open_key_ext: OCR error message : PROC-32: Cluster Ready Services on the local node is not running Messaging error [gipcretConnectionLost] [12]
2013-09-08 10:37:56.672: [ GPNP][6]clsgpnpco_ocr2profile: [at clsgpnpco.c:578] Result: (59) CLSGPNP_OCR_NOSRV. Failed to open requested OCR Profile.
2013-09-08 10:37:56.672: [ GPNP][6]clsgpnpd_reconcile: [at clsgpnpd.c:2449] Result: (59) CLSGPNP_OCR_NOSRV. Did not get a shared profile, OCR provider 60000000000123a0
2013-09-08 10:37:56.673: [ GPNP][6]clsgpnpm_sendMsg: [at clsgpnpm.c:3665] GIPC gipcretConnectionLost (12) gipcSendSync(ep,mbod) Msg-reply body mbuf=60000000010ad660 send failed mlen=373
2013-09-08 10:37:56.673: [ GPNP][6]clsgpnpd_dispatch: [at clsgpnpd.c:6035] Result: (31) CLSGPNP_CALL_AGAIN. Send response failed for msg=6000000000bf0320 to "tcp://10.15.34.11:57940"
2013-09-08 10:37:56.674: [ GPNP][6]clsgpnpm_sendResponseHttpErr: [at clsgpnpm.c:3730] GIPC gipcretConnectionLost (12) gipcSendSync(ep,hdr) Msg-err-reply hbuf=9fffffff5fd0cc20 send failed hlen=63/0: HTTP/1.0 400 Bad Request
Host: rac08
Server: GPnPD/1.0

2013-09-08 10:37:56.674: [ GPNP][6]clsgpnpd_dispatch: [at clsgpnpd.c:6074] Result: (48) CLSGPNP_COMM_ERR. Send error response failed for msg=6000000000bf0320 to "tcp://10.15.34.11:57940"
2013-09-08 10:37:56.674: [ default][6]Errors occurred while processing received gpnp message.

CRS-2332:Error pushing GPnP profile to "mdns:service:gpnp._tcp.local.://rac04:50587/agent=gpnpd,cname=ngprod_cluster,host=<HOSTNAME1>,pid=20308/gpnpd h:rac04 c:ngprod_cluster".

CRS-2340:Errors occurred while processing received gpnp message.