My Oracle Support Banner

EM 13c: Enterprise Manager 13c Cloud Control Agent topsql Monitoring Utilizes 25-30% CPU (Doc ID 2204645.1)

Last updated on SEPTEMBER 03, 2021

Applies to:

Enterprise Manager for Oracle Database - Version 11.2.0.1 and later
Information in this document applies to any platform.

Symptoms


The Enterprise Manager (EM) 13c Cloud Control agent CPU utilization spikes between 20% to 40% every 10 min and returns to idle again for 10 minutes.

The topSqlMonitoringList Collection consumes high CPU when the scheduled collection runs on this 10 minute cycle.


The next section will illustrate how to investigate high CPU consumption and which thread causes this issue:

  • The topas utility on AIX shows JAVA process 8716564 consumes high CPU, within a 20% to 40% range in a ten minute period.
  • To find the highest use thread in the JAVA process by run the following command:

            >> ps -mp 8716564 -o THREAD

  • The highest thread was 133103767, so change this number to hexadecimal "7EF0097" and search on this hexadecimal number in the the thread dump.

  • Generate a thread dump with one of these methods:
  1. Run jstack <pid>

  2. Kill -3 and the thread dump will be generated in the gcagent.log
  • Checking the 133103767 thread shows it is associated with the Top SQL Monitor Collection as follows:


                 3XMTHREADINFO "GC.Executor.73544 (oracle_database:ecomop1:topSqlMonitoringList_col) (oracle_database:<dbname>:topSqlMonitoringList_col:topSqlMonitoringList)" J9VMThread:0x0000000050ECB100, j9thread_t:0x00000100145F0220, java/lang/Thread:0x0000000042EBB490,
                 3XMJAVALTHREAD (java/lang/Thread getId:0x2B1CD, isDaemon:false)
                 3XMTHREADINFO1 (native thread ID:0x7EF0097, native priority:0x5, native policy:UNKNOWN, vmstate:CW, vm thread flags:0x00000001)
                 3XMCPUTIME CPU usage total: 65.576614000 secs, user: 56.259189000 secs, system: 9.317425000 secs
                 3XMHEAPALLOC Heap bytes allocated since last GC cycle=10837560 (0xA55E38)
                 3XMTHREADINFO3 Java callstack:
                 4XESTACKTRACE at oracle/sysman/gcagent/target/interaction/execution/Threshold$Key.toString(Threshold.java:984(Compiled Code))
                 4XESTACKTRACE at oracle/sysman/gcagent/target/interaction/execution/Threshold.mergeState(Threshold.java:2542(Compiled Code))
                 5XESTACKTRACE (entered lock: java/util/TreeMap@0x000000004217A848, entry count: 2)
                 4XESTACKTRACE at oracle/sysman/gcagent/target/interaction/execution/Threshold.mergeState(Threshold.java:2514(Compiled Code))
                 4XESTACKTRACE at oracle/sysman/gcagent/target/interaction/execution/Threshold.recordState(Threshold.java:2498(Compiled Code))
                 4XESTACKTRACE at oracle/sysman/gcagent/target/interaction/execution/Threshold.threshold_clearAll(Threshold.java:3196(Compiled Code))
                 5XESTACKTRACE (entered lock: java/util/TreeMap@0x000000004217A848, entry count: 1)
                 4XESTACKTRACE at oracle/sysman/gcagent/target/interaction/execution/CheckThreshold.checkThresholds(CheckThreshold.java:438(Compiled Code))
                 4XESTACKTRACE at oracle/sysman/gcagent/target/interaction/execution/ExecuteTask.processResultSet(ExecuteTask.java:2174(Compiled Code))
                 4XESTACKTRACE at oracle/sysman/gcagent/target/interaction/execution/ExecuteTask.runTask(ExecuteTask.java:3268(Compiled Code))
                 4XESTACKTRACE at oracle/sysman/gcagent/target/interaction/execution/ExecuteTask.call(ExecuteTask.java:4217(Compiled Code))
                 4XESTACKTRACE at oracle/sysman/gcagent/metadata/impl/collection/MetricColl$1.call(MetricColl.java:552(Compiled Code))
                 4XESTACKTRACE at oracle/sysman/gcagent/metadata/impl/collection/MetricColl$1.call(MetricColl.java:513(Compiled Code))
                 4XESTACKTRACE at oracle/sysman/gcagent/task/TaskFutureImpl$WrappedTask.accountedCall(TaskFutureImpl.java:600(Compiled Code))
                 4XESTACKTRACE at oracle/sysman/gcagent/task/TaskFutureImpl$WrappedTask.call(TaskFutureImpl.java:644(Compiled Code))
                 4XESTACKTRACE at java/util/concurrent/FutureTask.run(FutureTask.java:274(Compiled Code))

 





Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!

In this Document
Symptoms
Cause
Solution
References


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.