OCI : High CPU Utilization by dcs-agent 'dcs.agent.service.ft.collection.scheduler.FtCollectorScheduler.sche dule' FtCollectorScheduler (Doc ID 2523021.1)

Last updated on APRIL 29, 2020

Applies to:

Symptoms

OCI DB System and user observed that dcs-agent consuming high CPU.

# ps -ef | grep dcs-agent
root     38496 96289  0 08:45 pts/0    00:00:00 grep dcs-agent
root     56583     1  7 Feb28 ?        11:39:14 java -Xms128m -Xmx512m -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=512m -XX:+DisableExplicitGC -XX:ParallelGCThreads=4 -XX:+PrintGCDetails -XX:+PrintHeapAtGC -XX:+PrintTenuringDistribution -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -Xloggc:/opt/oracle/dcs/log/gc-dcs-agent-%t-%p.log -XX:+UseGCLogFileRotation
-XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10M -Doracle.security.jps.config=/opt/oracle/dcs/agent/jps-config.xml -jar
/opt/oracle/dcs/bin/dcs-agent-2.4.14.3.0-SNAPSHOT.jar server /opt/oracle/dcs/conf/dcs-agent.json

#top -p 56583
Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
Cpu(s): 27.3%us,  8.7%sy,  0.0%ni, 62.8%id,  1.0%wa,  0.0%hi,  0.2%si,  
0.0%st
Mem:  15135268k total, 13376220k used,  1759048k free,   387972k buffers
Swap:  2097148k total,   380724k used,  1716424k free,  3782028k cached

 PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
56583 root      20   0 2975m 340m  10m S 99.8  2.3 699:33.87 java


O Following thread is spinning the CPU ( output of #top -H -p PID)

Tasks:  68 total,   1 running,  67 sleeping,   0 stopped,   0 zombie
Cpu(s): 27.3%us,  8.7%sy,  0.0%ni, 62.8%id,  1.0%wa,  0.0%hi,  0.2%si,  0.0%st
Mem:  15135268k total, 13386280k used,  1748988k free,   387984k buffers
Swap:  2097148k total,   380724k used,  1716424k free,  3782468k cached

 PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
51258 root      20   0 2975m 340m  10m R 93.2  2.3 686:39.43 java   (Thread id :0xc83a)
56583 root      20   0 2975m 340m  10m S  0.0  2.3   0:00.00 java

O From jstack/Thread dump

"ft-collector-10" #8913 daemon prio=5 os_prio=0 tid=0x00007fe66c115800 nid=0xc83a runnable [0x00007fe637cfa000]
  java.lang.Thread.State: RUNNABLE
at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(Unknown Source)
- locked <0x00000000e3e20c68> (a java.util.concurrent.ConcurrentHashMap$ReservationNode)
at com.oracle.dcs.agent.service.ft.collection.FtCollectorHelperImpl$Cache.getGiCrsctlPath(FtCollectorHelperImpl.java:91)
at com.oracle.dcs.agent.service.ft.collection.FtCollectorHelperImpl.getGiCrsctlPath(FtCollectorHelperImpl.java:301)
at com.oracle.dcs.agent.service.ft.collection.FtCollectorHelperImpl.computeAsmInstances(FtCollectorHelperImpl.java:348)
at com.oracle.dcs.agent.service.ft.collection.FtCollectorHelperImpl.lambda$getAsmInstances$14(FtCollectorHelperImpl.java:367)
at com.oracle.dcs.agent.service.ft.collection.FtCollectorHelperImpl$$Lambda$156/1226465900.get(Unknown Source)
at com.oracle.dcs.agent.service.ft.collection.FtCollectorHelperImpl$Cache.lambda$getAsmInstances$7(FtCollectorHelperImpl.java:103)
at com.oracle.dcs.agent.service.ft.collection.FtCollectorHelperImpl$Cache$$Lambda$160/202958123.apply(Unknown Source)
at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(Unknown Source)
- locked <0x00000000e3e20c68> (a java.util.concurrent.ConcurrentHashMap$ReservationNode)
at com.oracle.dcs.agent.service.ft.collection.FtCollectorHelperImpl$Cache.getAsmInstances(FtCollectorHelperImpl.java:102)
at com.oracle.dcs.agent.service.ft.collection.FtCollectorHelperImpl.getAsmInstances(FtCollectorHelperImpl.java:365)
at com.oracle.dcs.agent.service.ft.collection.FtCollectorHelperImpl.computeIsAsmFirstInstance(FtCollectorHelperImpl.java:373)
at com.oracle.dcs.agent.service.ft.collection.FtCollectorHelperImpl.lambda$isAsmFirstInstance$15(FtCollectorHelperImpl.java:384)
at com.oracle.dcs.agent.service.ft.collection.FtCollectorHelperImpl$$Lambda$151/1138886940.get(Unknown Source)
at com.oracle.dcs.agent.service.ft.collection.FtCollectorHelperImpl$Cache.lambda$isAsmFirstInstance$8(FtCollectorHelperImpl.java:108)
at com.oracle.dcs.agent.service.ft.collection.FtCollectorHelperImpl$Cache$$Lambda$155/1294331664.apply(Unknown Source)
at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(Unknown Source)
- locked <0x00000000e3e20eb8> (a java.util.concurrent.ConcurrentHashMap$ReservationNode)
at com.oracle.dcs.agent.service.ft.collection.FtCollectorHelperImpl$Cache.isAsmFirstInstance(FtCollectorHelperImpl.java:107)
at com.oracle.dcs.agent.service.ft.collection.FtCollectorHelperImpl.isAsmFirstInstance(FtCollectorHelperImpl.java:382)
at com.oracle.dcs.agent.service.ft.collection.collectors.FtStorageCollector$FtStorageAsmCollector.doCollection(FtStorageCollector.java:360)
at com.oracle.dcs.agent.service.ft.collection.collectors.FtCollectorBase.collect(FtCollectorBase.java:56)
at com.oracle.dcs.agent.service.ft.collection.scheduler.FtCollectorScheduler$1.call(FtCollectorScheduler.java:69)
at com.oracle.dcs.agent.service.ft.collection.scheduler.FtCollectorScheduler$1.call(FtCollectorScheduler.java:66)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)


O dcs-agent version
#rpm -qa | grep dcs
dcs-agent-18.3.3.0.0_LINUX.X64_190129-128.x86_64  <<<< dcs-agent version
dcs-cli-18.3.3.0.0_LINUX.X64_190129-1.x86_64  
dcs-admin-18.1.1.0.0_LINUX.X64_180207-1.x86_64

Changes

Cause

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.