ASM fails to start with error 29702 as CSSD rejected grock DB+ASM change (Doc ID 1950918.1)

Last updated on JULY 22, 2015

Applies to:

Oracle Database - Enterprise Edition - Version 12.1.0.2 to 12.1.0.2 [Release 12.1]
Information in this document applies to any platform.

Symptoms

Grid Infrastructure setup, ASM fails to start during stack restart on node1:

<ORACLE_BASE>/diag/crs/<node>/crs/trace/alert.log

2014-11-08 11:08:49.114 [ORAAGENT(62198)]CRS-5017: The resource action "ora.asm start" encountered the following error:
2014-11-08 11:08:49.114+ORA-29702: error occurred in Cluster Group Service operation
. For details refer to "(:CLSN00107:)" in "/mount/oracle/gridbase/diag/crs/alpcissdb721/crs/trace/ohasd_oraagent_grid.trc".

 

alert_+ASM1.log

Starting background process MMNL
Sat Nov 08 11:08:46 2014
MMNL started with pid=25, OS id=65387
Sat Nov 08 11:08:46 2014
USER (ospid: 64192): terminating the instance due to error 29702
Sat Nov 08 11:08:47 2014
Instance terminated by USER, pid = 64192

 

+ASM1_lmon_<pid>.trc

kjxgrdmpcpu: High load threshold 5120
2014-11-08 11:08:46.664: [ CSSCLNT]clssgsGroupJoin: bad server response(-24)
kgxgnreg: clssgsreg: error: status 1 (0 ) group DB+ASM member 0
kjxgmjoin: can not join the group (DB+ASM) with id 0 (inst 1)
kjxgmjoin: kgxgn error 16
2014-11-08 11:08:46.664530 : kjxgrcln_dismount: closing IMR recording device
2014-11-08 11:08:46.666167 : kjxgrdisable: IMR recording device closed, terminating IMR
kjfmreg: Joining the cluster failed with err code 16

 

<ORACLE_BASE>/diag/crs/<node>/crs/trace/ocssd.trc

2014-11-08 11:08:46.660961 : CSSD:3064162624: clssgmpcJoinGrock: Processing join request(0x7fcca812b0cc) for grock(DB+ASM), member number(0), from clientID 1:34:1, msgseq(43) with 2 group attributes and 1 member attributes
2014-11-08 11:08:46.661148 : CSSD:3064162624: clssgmRPC: rpc 0x7fccb6bb8c10 (RPC#16) tag(10002a) sent to node 2
2014-11-08 11:08:46.663311 : CSSD:3065739584: clssgmRPCDone: rpc 0x7fccb6bb8c10 (RPC#16) state 6, flags 0x102
2014-11-08 11:08:46.663327 : CSSD:3065739584: clssgmAddGrockMemCmpl: Failed for clientID 1:34:1, rc -24
2014-11-08 11:08:46.663334 : CSSD:3065739584: clssgmAddGrockMemCmpl: sending response, status -24 for memberID 0:0:0:0 to clientID 1:34:1:1415462926, msg sequence 43
2014-11-08 11:08:46.663917 : CSSD:3058112832: clssgmcpJoinResp: Response with status -24 for clientID 1:34:1

 

<ORACLE_BASE>/diag/crs/<node>/crs/trace/ocssd.trc - node2 (stack been up for the longest time) 

2014-11-08 11:08:46.662352 : CSSD:3594422592: clssgmCheckGrpAttrCompat: The change was rejected due to incompatible killable states grock DB+ASM, ID 23586, current state 0, requested state 1
2014-11-08 11:08:46.662388 : CSSD:3594422592: clssgmCheckGrpAttrCompat: grock(DB+ASM), ID(23586), attributes checked(2)
2014-11-08 11:08:46.662404 : CSSD:3594422592: clssgmHandleGrockRcfgMaster: from node node1, 1, grock DB+ASM, RPC#16, tag 0x10002a, updateseq 0, operation sequence 0, status -24, send response 1

 

Another symptom when CSS crashes:

2015-01-29 11:28:10.146452 : CSSD:3633108736: clssgmVerifyGrock: grock(DBPAVUS) killable flag(0x1) out of sync with master(0x0)
2015-01-29 11:28:10.146458 : CSSD:3633108736: clssnmRemoveNodeInTerm: node 4, eagnmnmep1c23 terminated. Removing from its own member and connected bitmaps
2015-01-29 11:28:10.146483 : CSSD:3633108736: ###################################
2015-01-29 11:28:10.146486 : CSSD:3633108736: clssscExit: CSSD aborting from thread GM Peer Lsnr

 

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms