rootupgrade.sh fails as CRSD crashes after gipcretOutOfMemory (Doc ID 2095686.1)

Last updated on OCTOBER 14, 2016

Applies to:

Oracle Database - Standard Edition - Version 12.1.0.2 and later
Information in this document applies to any platform.

Symptoms

Upgrading Grid Infrastructure from 11.2.0.4 to 12.1.0.2, rootupgrade.sh on last node fails:

root script log

2016-01-10 16:28:40: Executing /oracle12c/app/12.1.0/grid/product/dbhome1/bin/crsctl startupgrade
2016-01-10 16:28:40: Executing cmd: /oracle12c/app/12.1.0/grid/product/dbhome1/bin/crsctl startupgrade
2016-01-10 16:29:04: Command output:
> Start upgrade invoked..
> CRS-4000: Command StartUpgrade failed, or completed with errors.
>End Command output
2016-01-10 16:29:14: Executing cmd: /oracle12c/app/12.1.0/grid/product/dbhome1/bin/crsctl check crs
2016-01-10 16:29:14: Command output:
> CRS-4638: Oracle High Availability Services is online
> CRS-4535: Cannot communicate with Cluster Ready Services
> CRS-4529: Cluster Synchronization Services is online
> CRS-4533: Event Manager is online
>End Command output
....
2016-01-10 16:29:25: Executing cmd: /oracle12c/app/12.1.0/grid/product/dbhome1/bin/clsecho -p has -f clsrsc -m 251
2016-01-10 16:29:25: Command output:
> CLSRSC-251: Oracle Clusterware stack start initiated but failed to complete
>End Command output

 

crsd.log from new home

...

CLSB:10538: Oracle Clusterware infrastructure error in CRSD (OS PID 4849812): Fatal signal 6 has occurred in program crsd thread 10538; nested signal count is 1
2016-01-10 16:29:03.968936 :GIPCXCPT:3342: gipclibMalloc: failed to allocate 4096 bytes, cowork 1127ad5d8, ret gipcretOutOfMemory (28) <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2016-01-10 16:29:03.969029 :GIPCXCPT:3342: gipcWaitF [clsssRecvMsg : clsss.c : 6983]: EXCEPTION[ ret gipcretOutOfMemory (28) ] failed to wait on obj 111214d50 [0000000000000486] { gipcEndpoint : localAddr 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=)(GIPCID=f1bdd962-000013e1-4849812))', remoteAddr 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_ctsdcuatdb2_)(GIPCID=000013e1-f1bdd962-4391076))', numPend 0, numReady 0, numDone 0, numDead 0, numTransfer 0, objFlags 0x0, pidPeer 4391076, readyRef 0, ready 0, wobj 112351490, sendp 0 status 0flags 0x20038616, flags-2 0x0, usrFlags 0x20010 }, reqList 1127adf70, nreq 1, creq 1127ae038 timeout 0 ms, flags 0x0
2016-01-10 16:29:03.969035 : CSSCLNT:3342: clsssRecvMsg: comm error received, ret 28, con (486), msg (1127ae238), msgl 310043200, response type 1
2016-01-10 16:29:03.969040 : CSSCLNT:3342: clssgsGroupGetStatus: communications failed (0/3/-1)
2016-01-10 16:29:03.969043 : CSSCLNT:3342: clssgsGroupGetStatus: returning 8
2016-01-10 16:29:03.969076 : OCRMAS:3342: th_monitor_ocrlocalgrp: Problem in clssgsgrpstat. CSS error code [8]
2016-01-10 16:29:03.969125 :GIPCXCPT:3342: gipclibMalloc: failed to allocate 4096 bytes, cowork 1127ac7c8, ret gipcretOutOfMemory (28) <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2016-01-10 16:29:03.969166 :GIPCXCPT:3342: gipcWaitF [gipcInternalSendSync : gipcInternal.c : 992]: EXCEPTION[ ret gipcretOutOfMemory (28) ] failed to wait on obj 111214d50 [0000000000000486] { gipcEndpoint : localAddr 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=)(GIPCID=f1bdd962-000013e1-4849812))', remoteAddr 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_ctsdcuatdb2_)(GIPCID=000013e1-f1bdd962-4391076))', numPend 1, numReady 0, numDone 0, numDead 0, numTransfer 0, objFlags 0x0, pidPeer 4391076, readyRef 0, ready 0, wobj 112351490, sendp 0 status 0flags 0x20038616, flags-2 0x0, usrFlags 0x20010 }, reqList 1127ad160, nreq 1, creq 1127ad228 timeout INFINITE, flags 0x8000100
2016-01-10 16:29:03.969220 :GIPCXCPT:3342: gipcSendSyncF [clsssServerRPC : clsss.c : 6791]: EXCEPTION[ ret gipcretOutOfMemory (28) ] failed to send on endp 111214d50 [0000000000000486] { gipcEndpoint : localAddr 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=)(GIPCID=f1bdd962-000013e1-4849812))', remoteAddr 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_ctsdcuatdb2_)(GIPCID=000013e1-f1bdd962-4391076))', numPend 1, numReady 0, numDone 0, numDead 0, numTransfer 0, objFlags 0x0, pidPeer 4391076, readyRef 0, ready 0, wobj 112351490, sendp 0 status 0flags 0x20038616, flags-2 0x0, usrFlags 0x20010 }, addr 0000000000000000, buf 1127ae048, len 80, flags 0x8000000
2016-01-10 16:29:03.969225 : CSSCLNT:3342: clsssServerRPC: send failed with err 28, msg type 7

2016-01-10 16:29:03.969228 : CSSCLNT:3342: clsssCommonClientExit: RPC failure, rc 3

2016-01-10 16:29:03.969308 : OCRAPI:3342: procr_ctx_set_invalid_no_abort: ctx is in state [10].
2016-01-10 16:29:03.969327 : OCRAPI:3342: procr_ctx_set_invalid_no_abort: ctx set to invalid
2016-01-10 16:29:03.969330 : OCRAPI:3342: procr_ctx_set_invalid_no_abort: Exiting...
2016-01-10 16:29:03.970510 :GIPCXCPT:6940: gipclibMalloc: failed to allocate 10392 bytes, cowork 113693f18, ret gipcretOutOfMemory (28) <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2016-01-10 16:29:03.970856 : CSSCLNT:6940: clsssRecvMsg: got a disconnect from the server while waiting for message type 1
2016-01-10 16:29:03.970869 : CSSCLNT:6940: clssgsGroupGetStatus: communications failed (0/3/-1)
2016-01-10 16:29:03.970872 : CSSCLNT:6940: clssgsGroupGetStatus: returning 8
2016-01-10 16:29:03.970903 : CRSEVT:6940: Error in clssgsgrpstat rc =8
Trace file /orabase12c/app/12.1.0/grid/diag/crs/ctsdcuatdb2/crs/trace/crsd.trc
Oracle Database 12c Clusterware Release 12.1.0.2.0 - Production Copyright 1996, 2014 Oracle. All rights reserved.
CLSB:1: Argument count (argc) for this daemon is 2
CLSB:1: Argument 0 is: /oracle12c/app/12.1.0/grid/product/dbhome1/bin/crsd.bin

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms