12c ocssd.bin threads using high CPU usage
(Doc ID 2239173.1)
Last updated on JULY 30, 2023
Applies to:
Oracle Database - Enterprise Edition - Version 12.1.0.2 and laterOracle Database Exadata Express Cloud Service - Version N/A and later
Oracle Cloud Infrastructure - Database Service - Version N/A and later
Oracle Database Cloud Exadata Service - Version N/A and later
Oracle Database Backup Service - Version N/A and later
Information in this document applies to any platform.
Symptoms
12.1.0.2 Grid Infrastructure cluster, ocssd.bin some threads as highlighted are seen to be using high CPU on one node:
on Linux use cmd: ps -Lo s,lwp,class,pri,psr,tty,user,pcpu,time,pmem,vsz,rss,args --sort=pcpu -p pid-of-ocssd.bin
S LWP CLS PRI PSR TT USER %CPU TIME %MEM VSZ RSS COMMAND
S 25477 RR 139 1 ? oracle 0.0 00:00:00 0.0 2508964 247220 /u01/app/12.1.0.2/grid/bin/ocssd.bin
S 25486 RR 139 1 ? oracle 0.2 01:42:04 0.0 2508964 247220 /u01/app/12.1.0.2/grid/bin/ocssd.bin
S 25489 RR 139 10 ? oracle 0.0 00:08:25 0.0 2508964 247220 /u01/app/12.1.0.2/grid/bin/ocssd.bin
S 25553 RR 139 12 ? oracle 0.0 00:00:00 0.0 2508964 247220 /u01/app/12.1.0.2/grid/bin/ocssd.bin
S 25554 RR 139 7 ? oracle 0.0 00:00:00 0.0 2508964 247220 /u01/app/12.1.0.2/grid/bin/ocssd.bin
S 25555 RR 139 12 ? oracle 27.0 8-13:42:26 0.0 2508964 247220 /u01/app/12.1.0.2/grid/bin/ocssd.bin ====>>>>
S 25556 RR 139 14 ? oracle 26.3 8-08:44:16 0.0 2508964 247220 /u01/app/12.1.0.2/grid/bin/ocssd.bin ====>>>>
S 25557 RR 139 13 ? oracle 0.0 00:00:00 0.0 2508964 247220 /u01/app/12.1.0.2/grid/bin/ocssd.bin
S 25558 RR 139 5 ? oracle 0.0 00:00:00 0.0 2508964 247220 /u01/app/12.1.0.2/grid/bin/ocssd.bin
S 25559 RR 139 15 ? oracle 0.0 00:10:38 0.0 2508964 247220 /u01/app/12.1.0.2/grid/bin/ocssd.bin
S 25560 RR 139 2 ? oracle 5.6 1-18:35:37 0.0 2508964 247220 /u01/app/12.1.0.2/grid/bin/ocssd.bin
R 25561 RR 139 8 ? oracle 26.2 8-07:41:52 0.0 2508964 247220 /u01/app/12.1.0.2/grid/bin/ocssd.bin ====>>
S 25562 RR 139 6 ? oracle 0.0 00:00:04 0.0 2508964 247220 /u01/app/12.1.0.2/grid/bin/ocssd.bin
S 25563 RR 139 13 ? oracle 5.6 1-19:03:19 0.0 2508964 247220 /u01/app/12.1.0.2/grid/bin/ocssd.bin
S 25477 RR 139 1 ? oracle 0.0 00:00:00 0.0 2508964 247220 /u01/app/12.1.0.2/grid/bin/ocssd.bin
S 25486 RR 139 1 ? oracle 0.2 01:42:04 0.0 2508964 247220 /u01/app/12.1.0.2/grid/bin/ocssd.bin
S 25489 RR 139 10 ? oracle 0.0 00:08:25 0.0 2508964 247220 /u01/app/12.1.0.2/grid/bin/ocssd.bin
S 25553 RR 139 12 ? oracle 0.0 00:00:00 0.0 2508964 247220 /u01/app/12.1.0.2/grid/bin/ocssd.bin
S 25554 RR 139 7 ? oracle 0.0 00:00:00 0.0 2508964 247220 /u01/app/12.1.0.2/grid/bin/ocssd.bin
S 25555 RR 139 12 ? oracle 27.0 8-13:42:26 0.0 2508964 247220 /u01/app/12.1.0.2/grid/bin/ocssd.bin ====>>>>
S 25556 RR 139 14 ? oracle 26.3 8-08:44:16 0.0 2508964 247220 /u01/app/12.1.0.2/grid/bin/ocssd.bin ====>>>>
S 25557 RR 139 13 ? oracle 0.0 00:00:00 0.0 2508964 247220 /u01/app/12.1.0.2/grid/bin/ocssd.bin
S 25558 RR 139 5 ? oracle 0.0 00:00:00 0.0 2508964 247220 /u01/app/12.1.0.2/grid/bin/ocssd.bin
S 25559 RR 139 15 ? oracle 0.0 00:10:38 0.0 2508964 247220 /u01/app/12.1.0.2/grid/bin/ocssd.bin
S 25560 RR 139 2 ? oracle 5.6 1-18:35:37 0.0 2508964 247220 /u01/app/12.1.0.2/grid/bin/ocssd.bin
R 25561 RR 139 8 ? oracle 26.2 8-07:41:52 0.0 2508964 247220 /u01/app/12.1.0.2/grid/bin/ocssd.bin ====>>
S 25562 RR 139 6 ? oracle 0.0 00:00:04 0.0 2508964 247220 /u01/app/12.1.0.2/grid/bin/ocssd.bin
S 25563 RR 139 13 ? oracle 5.6 1-19:03:19 0.0 2508964 247220 /u01/app/12.1.0.2/grid/bin/ocssd.bin
The stack for those threads using cmd pstack shows:
Thread 30 (Thread 0x7fe97be7e700 (LWP 25555)):
#0 0x00007fe98907a0a3 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fe98f66c370 in sgipcwEpollWaitHelper () from /u01/app/12.1.0.2/grid/lib/libhasgen12.so
#2 0x00007fe98f666b90 in sgipcwWait () from /u01/app/12.1.0.2/grid/lib/libhasgen12.so
#3 0x00007fe98f3c3039 in gipcWaitOsd () from /u01/app/12.1.0.2/grid/lib/libhasgen12.so
#4 0x00007fe98f3acf2a in gipcInternalWaitEpoll () from /u01/app/12.1.0.2/grid/lib/libhasgen12.so
#5 0x00007fe98f3a633b in gipcInternalWait () from /u01/app/12.1.0.2/grid/lib/libhasgen12.so
#6 0x00007fe98f33a2bd in gipcWaitF () from /u01/app/12.1.0.2/grid/lib/libhasgen12.so
#7 0x0000000000567558 in clssbcm_worker ()
#8 0x000000000054bf4c in clssscthrdmain ()
#9 0x00007fe98a9eaaa1 in start_thread () from /lib64/libpthread.so.0
#10 0x00007fe989079aad in clone () from /lib64/libc.so.6
#0 0x00007fe98907a0a3 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fe98f66c370 in sgipcwEpollWaitHelper () from /u01/app/12.1.0.2/grid/lib/libhasgen12.so
#2 0x00007fe98f666b90 in sgipcwWait () from /u01/app/12.1.0.2/grid/lib/libhasgen12.so
#3 0x00007fe98f3c3039 in gipcWaitOsd () from /u01/app/12.1.0.2/grid/lib/libhasgen12.so
#4 0x00007fe98f3acf2a in gipcInternalWaitEpoll () from /u01/app/12.1.0.2/grid/lib/libhasgen12.so
#5 0x00007fe98f3a633b in gipcInternalWait () from /u01/app/12.1.0.2/grid/lib/libhasgen12.so
#6 0x00007fe98f33a2bd in gipcWaitF () from /u01/app/12.1.0.2/grid/lib/libhasgen12.so
#7 0x0000000000567558 in clssbcm_worker ()
#8 0x000000000054bf4c in clssscthrdmain ()
#9 0x00007fe98a9eaaa1 in start_thread () from /lib64/libpthread.so.0
#10 0x00007fe989079aad in clone () from /lib64/libc.so.6
Thread 29 (Thread 0x7fe97bcfd700 (LWP 25556)):
#0 0x00007fe98907a0a3 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fe98f66c370 in sgipcwEpollWaitHelper () from /u01/app/12.1.0.2/grid/lib/libhasgen12.so
#2 0x00007fe98f666b90 in sgipcwWait () from /u01/app/12.1.0.2/grid/lib/libhasgen12.so
#3 0x00007fe98f3c3039 in gipcWaitOsd () from /u01/app/12.1.0.2/grid/lib/libhasgen12.so
#4 0x00007fe98f3acf2a in gipcInternalWaitEpoll () from /u01/app/12.1.0.2/grid/lib/libhasgen12.so
#5 0x00007fe98f3a633b in gipcInternalWait () from /u01/app/12.1.0.2/grid/lib/libhasgen12.so
#6 0x00007fe98f33a2bd in gipcWaitF () from /u01/app/12.1.0.2/grid/lib/libhasgen12.so
#7 0x0000000000567558 in clssbcm_worker ()
#8 0x000000000054bf4c in clssscthrdmain ()
#9 0x00007fe98a9eaaa1 in start_thread () from /lib64/libpthread.so.0
#10 0x00007fe989079aad in clone () from /lib64/libc.so.6
#0 0x00007fe98907a0a3 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fe98f66c370 in sgipcwEpollWaitHelper () from /u01/app/12.1.0.2/grid/lib/libhasgen12.so
#2 0x00007fe98f666b90 in sgipcwWait () from /u01/app/12.1.0.2/grid/lib/libhasgen12.so
#3 0x00007fe98f3c3039 in gipcWaitOsd () from /u01/app/12.1.0.2/grid/lib/libhasgen12.so
#4 0x00007fe98f3acf2a in gipcInternalWaitEpoll () from /u01/app/12.1.0.2/grid/lib/libhasgen12.so
#5 0x00007fe98f3a633b in gipcInternalWait () from /u01/app/12.1.0.2/grid/lib/libhasgen12.so
#6 0x00007fe98f33a2bd in gipcWaitF () from /u01/app/12.1.0.2/grid/lib/libhasgen12.so
#7 0x0000000000567558 in clssbcm_worker ()
#8 0x000000000054bf4c in clssscthrdmain ()
#9 0x00007fe98a9eaaa1 in start_thread () from /lib64/libpthread.so.0
#10 0x00007fe989079aad in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7fe97b135700 (LWP 25561)):
#0 0x00007fe98907a0a3 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fe98f66c370 in sgipcwEpollWaitHelper () from /u01/app/12.1.0.2/grid/lib/libhasgen12.so
#2 0x00007fe98f666b90 in sgipcwWait () from /u01/app/12.1.0.2/grid/lib/libhasgen12.so
#3 0x00007fe98f3c3039 in gipcWaitOsd () from /u01/app/12.1.0.2/grid/lib/libhasgen12.so
#4 0x00007fe98f3acf2a in gipcInternalWaitEpoll () from /u01/app/12.1.0.2/grid/lib/libhasgen12.so
#5 0x00007fe98f3a633b in gipcInternalWait () from /u01/app/12.1.0.2/grid/lib/libhasgen12.so
#6 0x00007fe98f33a2bd in gipcWaitF () from /u01/app/12.1.0.2/grid/lib/libhasgen12.so
#7 0x0000000000550490 in clssscSelect ()
#8 0x00000000004eea4b in clssgmProcClientReqs ()
#9 0x00000000004ee71a in clssgmclientlsnr ()
#10 0x000000000054bf4c in clssscthrdmain ()
#11 0x00007fe98a9eaaa1 in start_thread () from /lib64/libpthread.so.0
#12 0x00007fe989079aad in clone () from /lib64/libc.so.6
#0 0x00007fe98907a0a3 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fe98f66c370 in sgipcwEpollWaitHelper () from /u01/app/12.1.0.2/grid/lib/libhasgen12.so
#2 0x00007fe98f666b90 in sgipcwWait () from /u01/app/12.1.0.2/grid/lib/libhasgen12.so
#3 0x00007fe98f3c3039 in gipcWaitOsd () from /u01/app/12.1.0.2/grid/lib/libhasgen12.so
#4 0x00007fe98f3acf2a in gipcInternalWaitEpoll () from /u01/app/12.1.0.2/grid/lib/libhasgen12.so
#5 0x00007fe98f3a633b in gipcInternalWait () from /u01/app/12.1.0.2/grid/lib/libhasgen12.so
#6 0x00007fe98f33a2bd in gipcWaitF () from /u01/app/12.1.0.2/grid/lib/libhasgen12.so
#7 0x0000000000550490 in clssscSelect ()
#8 0x00000000004eea4b in clssgmProcClientReqs ()
#9 0x00000000004ee71a in clssgmclientlsnr ()
#10 0x000000000054bf4c in clssscthrdmain ()
#11 0x00007fe98a9eaaa1 in start_thread () from /lib64/libpthread.so.0
#12 0x00007fe989079aad in clone () from /lib64/libc.so.6
ocssd.trc is normal and not much difference between nodes:
2017-01-12 05:01:24.269286 : CSSD:31332096: clssnmSendingThread: sending status msg to all nodes
2017-01-12 05:01:24.269391 : CSSD:31332096: clssnmSendingThread: sent 5 status msgs to all nodes
2017-01-12 05:01:24.627162 : CSSD:945293056: clssgmMbrDataUpdt: Processing member data change type 1, size 4 for group HB+ASM, memberID 11:2:1
2017-01-12 05:01:24.627175 : CSSD:945293056: clssgmMbrDataUpdt: Sending member data change to GMP for group HB+ASM, memberID 11:2:1
2017-01-12 05:01:24.627465 : CSSD:948188928: clssgmpcMemberDataUpdt: grockName HB+ASM memberID 11:2:1, datatype 1 datasize 4
2017-01-12 05:01:24.628249 : CSSD:942139136: clssgmcpDataUpdtCmpl: Status 0 mbr data updt memberID 11:2:1 from clientID 1:38:4
2017-01-12 05:01:26.681497 : CSSD:945293056: clssgmMbrDataUpdt: Processing member data change type 1, size 4 for group HB+ASM, memberID 11:2:1
2017-01-12 05:01:26.681516 : CSSD:945293056: clssgmMbrDataUpdt: Sending member data change to GMP for group HB+ASM, memberID 11:2:1
2017-01-12 05:01:26.681768 : CSSD:948188928: clssgmpcMemberDataUpdt: grockName HB+ASM memberID 11:2:1, datatype 1 datasize 4
2017-01-12 05:01:26.682478 : CSSD:942139136: clssgmcpDataUpdtCmpl: Status 0 mbr data updt memberID 11:2:1 from clientID 1:38:4
2017-01-12 05:01:28.733801 : CSSD:945293056: clssgmMbrDataUpdt: Processing member data change type 1, size 4 for group HB+ASM, memberID 11:2:1
2017-01-12 05:01:28.733814 : CSSD:945293056: clssgmMbrDataUpdt: Sending member data change to GMP for group HB+ASM, memberID 11:2:1
2017-01-12 05:01:28.734153 : CSSD:948188928: clssgmpcMemberDataUpdt: grockName HB+ASM memberID 11:2:1, datatype 1 datasize 4
2017-01-12 05:01:28.734721 : CSSD:942139136: clssgmcpDataUpdtCmpl: Status 0 mbr data updt memberID 11:2:1 from clientID 1:38:4
2017-01-12 05:01:29.241472 : CSSD:945293056: clssgmClientConnectMsg: Connect from con(0x12f400f3) proc(0x7fcb12b4a890) pid(170526/170526) version 12:1:1:4, clientID 1:106307:0, msg flags 0x0000, properties: 1,2,3,4,5,6,7
2017-01-12 05:01:29.243377 : CSSD:945293056: clssgmcShareCreate: (0x7fcb13ddeb30) Same group share, by clientID 1:106307:1, with target group DG_LOCAL_DATA_MO01, memberID 2147483655:2:3
2017-01-12 05:01:29.243390 : CSSD:945293056: clssgmRegisterShared: local grock DG_LOCAL_DATA_MO01 member 3 share type 1 for clientID 1:106307:1 (0x7fcb11699880)
2017-01-12 05:01:29.244264 : CSSD:945293056: clssgmcShareCreate: (0x7fcb12fecda0) Same group share, by clientID 1:106307:2, with target group DBCLRREL, memberID 2615:2:0
2017-01-12 05:01:29.244275 : CSSD:945293056: clssgmRegisterShared: global grock DBCLRREL member 0 share type 1 for clientID 1:106307:2 (0x7fcb12fec6c0)
2017-01-12 05:01:29.270721 : CSSD:31332096: clssnmSendingThread: sending status msg to all nodes
2017-01-12 05:01:29.270810 : CSSD:31332096: clssnmSendingThread: sent 5 status msgs to all nodes
2017-01-12 05:01:24.269391 : CSSD:31332096: clssnmSendingThread: sent 5 status msgs to all nodes
2017-01-12 05:01:24.627162 : CSSD:945293056: clssgmMbrDataUpdt: Processing member data change type 1, size 4 for group HB+ASM, memberID 11:2:1
2017-01-12 05:01:24.627175 : CSSD:945293056: clssgmMbrDataUpdt: Sending member data change to GMP for group HB+ASM, memberID 11:2:1
2017-01-12 05:01:24.627465 : CSSD:948188928: clssgmpcMemberDataUpdt: grockName HB+ASM memberID 11:2:1, datatype 1 datasize 4
2017-01-12 05:01:24.628249 : CSSD:942139136: clssgmcpDataUpdtCmpl: Status 0 mbr data updt memberID 11:2:1 from clientID 1:38:4
2017-01-12 05:01:26.681497 : CSSD:945293056: clssgmMbrDataUpdt: Processing member data change type 1, size 4 for group HB+ASM, memberID 11:2:1
2017-01-12 05:01:26.681516 : CSSD:945293056: clssgmMbrDataUpdt: Sending member data change to GMP for group HB+ASM, memberID 11:2:1
2017-01-12 05:01:26.681768 : CSSD:948188928: clssgmpcMemberDataUpdt: grockName HB+ASM memberID 11:2:1, datatype 1 datasize 4
2017-01-12 05:01:26.682478 : CSSD:942139136: clssgmcpDataUpdtCmpl: Status 0 mbr data updt memberID 11:2:1 from clientID 1:38:4
2017-01-12 05:01:28.733801 : CSSD:945293056: clssgmMbrDataUpdt: Processing member data change type 1, size 4 for group HB+ASM, memberID 11:2:1
2017-01-12 05:01:28.733814 : CSSD:945293056: clssgmMbrDataUpdt: Sending member data change to GMP for group HB+ASM, memberID 11:2:1
2017-01-12 05:01:28.734153 : CSSD:948188928: clssgmpcMemberDataUpdt: grockName HB+ASM memberID 11:2:1, datatype 1 datasize 4
2017-01-12 05:01:28.734721 : CSSD:942139136: clssgmcpDataUpdtCmpl: Status 0 mbr data updt memberID 11:2:1 from clientID 1:38:4
2017-01-12 05:01:29.241472 : CSSD:945293056: clssgmClientConnectMsg: Connect from con(0x12f400f3) proc(0x7fcb12b4a890) pid(170526/170526) version 12:1:1:4, clientID 1:106307:0, msg flags 0x0000, properties: 1,2,3,4,5,6,7
2017-01-12 05:01:29.243377 : CSSD:945293056: clssgmcShareCreate: (0x7fcb13ddeb30) Same group share, by clientID 1:106307:1, with target group DG_LOCAL_DATA_MO01, memberID 2147483655:2:3
2017-01-12 05:01:29.243390 : CSSD:945293056: clssgmRegisterShared: local grock DG_LOCAL_DATA_MO01 member 3 share type 1 for clientID 1:106307:1 (0x7fcb11699880)
2017-01-12 05:01:29.244264 : CSSD:945293056: clssgmcShareCreate: (0x7fcb12fecda0) Same group share, by clientID 1:106307:2, with target group DBCLRREL, memberID 2615:2:0
2017-01-12 05:01:29.244275 : CSSD:945293056: clssgmRegisterShared: global grock DBCLRREL member 0 share type 1 for clientID 1:106307:2 (0x7fcb12fec6c0)
2017-01-12 05:01:29.270721 : CSSD:31332096: clssnmSendingThread: sending status msg to all nodes
2017-01-12 05:01:29.270810 : CSSD:31332096: clssnmSendingThread: sent 5 status msgs to all nodes
Cause
To view full details, sign in with your My Oracle Support account. |
|
Don't have a My Oracle Support account? Click to get started! |
In this Document
| Symptoms |
| Cause |
| Solution |
| References |