Ocssd.bin using High Memory Cause Node Reboot if Large Number of Trace Log in <Oracle Base>/diag/crs/<node name>/crs/trace Directory

(Doc ID 2310600.1)

Last updated on OCTOBER 02, 2017

Applies to:

Oracle Database - Enterprise Edition - Version 12.1.0.2 to 12.1.0.2 [Release 12.1]
Information in this document applies to any platform.

Symptoms

Clusterware alert log indicate css daemon terminating due to fatal error

2017-07-11 23:23:01.817 [OCSSD(25291)]CRS-1656: The CSS daemon is terminating due to a fatal error; Details at (:CSSSC00012:) in /oracle/app/grid/diag/crs/node1/crs/trace/ocssd.trc <<<<<<<<<<< cssd failed.

2017-07-11 23:23:01.894 [OCSSD(25291)]CRS-1652: Starting clean up of CRSD resources.

Further review shows that: 

1. Node reboot as ocssd can not allocate memory (From ocssd.trc) just prior to css termination:

  

2017-07-11 23:23:01.790310 :GIPCXCPT:16: gipclibMalloc: failed to allocate 40960 bytes, cowork 9ffffffffefcec70, ret gipcretOutOfMemory (28) <<<<<<<  Failed to allocate 40960 bytes
2017-07-11 23:23:01.790437 :GIPCXCPT:16: gipcWaitF [gipcInternalSendSync : gipcInternal.c : 1015]: EXCEPTION[ ret gipcretOutOfMemory (28) ] failed to wait on obj 6000
000000a7f3c0 [0000000000000466] { gipcEndpoint : localAddr 'udp://172.32.222.149:12041', remoteAddr '', numPend 5, numReady 1, numDone 0, numDead 0, numTransfer 0, obj
Flags 0x0, pidPeer 0, readyRef 0000000000000000, ready 0, wobj 6000000001de9da0, sendp 6000000001e11110 status 13flags 0x20000003, flags-2 0x80, usrFlags 0x4000 }, req
List 9ffffffffefcf6c0, nreq 1, creq 9ffffffffefcf6b0 timeout INFINITE, flags 0x100
 

2. Top output from OSWatcher shows that ocssd.bin use more than 2G memory during the time: 

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms