12c ocssd.bin threads using high CPU usage

(Doc ID 2239173.1)

Last updated on MAY 23, 2018

Applies to:

Oracle Database - Enterprise Edition - Version 12.1.0.2 and later
Information in this document applies to any platform.

Symptoms

12.1.0.2 Grid Infrastructure cluster, ocssd.bin some threads as highlighted are seen to be using high CPU on one node: 

on Linux use cmd: ps -Lo s,lwp,class,pri,psr,tty,user,pcpu,time,pmem,vsz,rss,args --sort=pcpu -p pid-of-ocssd.bin
S LWP CLS PRI PSR TT USER %CPU TIME %MEM VSZ RSS COMMAND
S 25477 RR 139 1 ? oracle 0.0 00:00:00 0.0 2508964 247220 /u01/app/12.1.0.2/grid/bin/ocssd.bin
S 25486 RR 139 1 ? oracle 0.2 01:42:04 0.0 2508964 247220 /u01/app/12.1.0.2/grid/bin/ocssd.bin
S 25489 RR 139 10 ? oracle 0.0 00:08:25 0.0 2508964 247220 /u01/app/12.1.0.2/grid/bin/ocssd.bin
S 25553 RR 139 12 ? oracle 0.0 00:00:00 0.0 2508964 247220 /u01/app/12.1.0.2/grid/bin/ocssd.bin
S 25554 RR 139 7 ? oracle 0.0 00:00:00 0.0 2508964 247220 /u01/app/12.1.0.2/grid/bin/ocssd.bin
S 25555 RR 139 12 ? oracle 27.0 8-13:42:26 0.0 2508964 247220 /u01/app/12.1.0.2/grid/bin/ocssd.bin ====>>>>
S 25556 RR 139 14 ? oracle 26.3 8-08:44:16 0.0 2508964 247220 /u01/app/12.1.0.2/grid/bin/ocssd.bin ====>>>>
S 25557 RR 139 13 ? oracle 0.0 00:00:00 0.0 2508964 247220 /u01/app/12.1.0.2/grid/bin/ocssd.bin
S 25558 RR 139 5 ? oracle 0.0 00:00:00 0.0 2508964 247220 /u01/app/12.1.0.2/grid/bin/ocssd.bin
S 25559 RR 139 15 ? oracle 0.0 00:10:38 0.0 2508964 247220 /u01/app/12.1.0.2/grid/bin/ocssd.bin
S 25560 RR 139 2 ? oracle 5.6 1-18:35:37 0.0 2508964 247220 /u01/app/12.1.0.2/grid/bin/ocssd.bin
R 25561 RR 139 8 ? oracle 26.2 8-07:41:52 0.0 2508964 247220 /u01/app/12.1.0.2/grid/bin/ocssd.bin ====>>
S 25562 RR 139 6 ? oracle 0.0 00:00:04 0.0 2508964 247220 /u01/app/12.1.0.2/grid/bin/ocssd.bin
S 25563 RR 139 13 ? oracle 5.6 1-19:03:19 0.0 2508964 247220 /u01/app/12.1.0.2/grid/bin/ocssd.bin

The stack for those threads using cmd pstack shows:

Thread 30 (Thread 0x7fe97be7e700 (LWP 25555)):
#0 0x00007fe98907a0a3 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fe98f66c370 in sgipcwEpollWaitHelper () from /u01/app/12.1.0.2/grid/lib/libhasgen12.so
#2 0x00007fe98f666b90 in sgipcwWait () from /u01/app/12.1.0.2/grid/lib/libhasgen12.so
#3 0x00007fe98f3c3039 in gipcWaitOsd () from /u01/app/12.1.0.2/grid/lib/libhasgen12.so
#4 0x00007fe98f3acf2a in gipcInternalWaitEpoll () from /u01/app/12.1.0.2/grid/lib/libhasgen12.so
#5 0x00007fe98f3a633b in gipcInternalWait () from /u01/app/12.1.0.2/grid/lib/libhasgen12.so
#6 0x00007fe98f33a2bd in gipcWaitF () from /u01/app/12.1.0.2/grid/lib/libhasgen12.so
#7 0x0000000000567558 in clssbcm_worker ()
#8 0x000000000054bf4c in clssscthrdmain ()
#9 0x00007fe98a9eaaa1 in start_thread () from /lib64/libpthread.so.0
#10 0x00007fe989079aad in clone () from /lib64/libc.so.6

 

Thread 29 (Thread 0x7fe97bcfd700 (LWP 25556)):
#0 0x00007fe98907a0a3 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fe98f66c370 in sgipcwEpollWaitHelper () from /u01/app/12.1.0.2/grid/lib/libhasgen12.so
#2 0x00007fe98f666b90 in sgipcwWait () from /u01/app/12.1.0.2/grid/lib/libhasgen12.so
#3 0x00007fe98f3c3039 in gipcWaitOsd () from /u01/app/12.1.0.2/grid/lib/libhasgen12.so
#4 0x00007fe98f3acf2a in gipcInternalWaitEpoll () from /u01/app/12.1.0.2/grid/lib/libhasgen12.so
#5 0x00007fe98f3a633b in gipcInternalWait () from /u01/app/12.1.0.2/grid/lib/libhasgen12.so
#6 0x00007fe98f33a2bd in gipcWaitF () from /u01/app/12.1.0.2/grid/lib/libhasgen12.so
#7 0x0000000000567558 in clssbcm_worker ()
#8 0x000000000054bf4c in clssscthrdmain ()
#9 0x00007fe98a9eaaa1 in start_thread () from /lib64/libpthread.so.0
#10 0x00007fe989079aad in clone () from /lib64/libc.so.6

 

Thread 24 (Thread 0x7fe97b135700 (LWP 25561)):
#0 0x00007fe98907a0a3 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fe98f66c370 in sgipcwEpollWaitHelper () from /u01/app/12.1.0.2/grid/lib/libhasgen12.so
#2 0x00007fe98f666b90 in sgipcwWait () from /u01/app/12.1.0.2/grid/lib/libhasgen12.so
#3 0x00007fe98f3c3039 in gipcWaitOsd () from /u01/app/12.1.0.2/grid/lib/libhasgen12.so
#4 0x00007fe98f3acf2a in gipcInternalWaitEpoll () from /u01/app/12.1.0.2/grid/lib/libhasgen12.so
#5 0x00007fe98f3a633b in gipcInternalWait () from /u01/app/12.1.0.2/grid/lib/libhasgen12.so
#6 0x00007fe98f33a2bd in gipcWaitF () from /u01/app/12.1.0.2/grid/lib/libhasgen12.so
#7 0x0000000000550490 in clssscSelect ()
#8 0x00000000004eea4b in clssgmProcClientReqs ()
#9 0x00000000004ee71a in clssgmclientlsnr ()
#10 0x000000000054bf4c in clssscthrdmain ()
#11 0x00007fe98a9eaaa1 in start_thread () from /lib64/libpthread.so.0
#12 0x00007fe989079aad in clone () from /lib64/libc.so.6

 

ocssd.trc is normal and not much difference between nodes:

2017-01-12 05:01:24.269286 : CSSD:31332096: clssnmSendingThread: sending status msg to all nodes
2017-01-12 05:01:24.269391 : CSSD:31332096: clssnmSendingThread: sent 5 status msgs to all nodes
2017-01-12 05:01:24.627162 : CSSD:945293056: clssgmMbrDataUpdt: Processing member data change type 1, size 4 for group HB+ASM, memberID 11:2:1
2017-01-12 05:01:24.627175 : CSSD:945293056: clssgmMbrDataUpdt: Sending member data change to GMP for group HB+ASM, memberID 11:2:1
2017-01-12 05:01:24.627465 : CSSD:948188928: clssgmpcMemberDataUpdt: grockName HB+ASM memberID 11:2:1, datatype 1 datasize 4
2017-01-12 05:01:24.628249 : CSSD:942139136: clssgmcpDataUpdtCmpl: Status 0 mbr data updt memberID 11:2:1 from clientID 1:38:4
2017-01-12 05:01:26.681497 : CSSD:945293056: clssgmMbrDataUpdt: Processing member data change type 1, size 4 for group HB+ASM, memberID 11:2:1
2017-01-12 05:01:26.681516 : CSSD:945293056: clssgmMbrDataUpdt: Sending member data change to GMP for group HB+ASM, memberID 11:2:1
2017-01-12 05:01:26.681768 : CSSD:948188928: clssgmpcMemberDataUpdt: grockName HB+ASM memberID 11:2:1, datatype 1 datasize 4
2017-01-12 05:01:26.682478 : CSSD:942139136: clssgmcpDataUpdtCmpl: Status 0 mbr data updt memberID 11:2:1 from clientID 1:38:4
2017-01-12 05:01:28.733801 : CSSD:945293056: clssgmMbrDataUpdt: Processing member data change type 1, size 4 for group HB+ASM, memberID 11:2:1
2017-01-12 05:01:28.733814 : CSSD:945293056: clssgmMbrDataUpdt: Sending member data change to GMP for group HB+ASM, memberID 11:2:1
2017-01-12 05:01:28.734153 : CSSD:948188928: clssgmpcMemberDataUpdt: grockName HB+ASM memberID 11:2:1, datatype 1 datasize 4
2017-01-12 05:01:28.734721 : CSSD:942139136: clssgmcpDataUpdtCmpl: Status 0 mbr data updt memberID 11:2:1 from clientID 1:38:4
2017-01-12 05:01:29.241472 : CSSD:945293056: clssgmClientConnectMsg: Connect from con(0x12f400f3) proc(0x7fcb12b4a890) pid(170526/170526) version 12:1:1:4, clientID 1:106307:0, msg flags 0x0000, properties: 1,2,3,4,5,6,7
2017-01-12 05:01:29.243377 : CSSD:945293056: clssgmcShareCreate: (0x7fcb13ddeb30) Same group share, by clientID 1:106307:1, with target group DG_LOCAL_DATA_MO01, memberID 2147483655:2:3
2017-01-12 05:01:29.243390 : CSSD:945293056: clssgmRegisterShared: local grock DG_LOCAL_DATA_MO01 member 3 share type 1 for clientID 1:106307:1 (0x7fcb11699880)
2017-01-12 05:01:29.244264 : CSSD:945293056: clssgmcShareCreate: (0x7fcb12fecda0) Same group share, by clientID 1:106307:2, with target group DBCLRREL, memberID 2615:2:0
2017-01-12 05:01:29.244275 : CSSD:945293056: clssgmRegisterShared: global grock DBCLRREL member 0 share type 1 for clientID 1:106307:2 (0x7fcb12fec6c0)
2017-01-12 05:01:29.270721 : CSSD:31332096: clssnmSendingThread: sending status msg to all nodes
2017-01-12 05:01:29.270810 : CSSD:31332096: clssnmSendingThread: sent 5 status msgs to all nodes

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms