EM 13.4: OMS Java Process Consuming High CPU and Causing Console Slowness / OMS Hung Intermittently
(Doc ID 2702019.1)
Last updated on NOVEMBER 19, 2020
Applies to:
Enterprise Manager Base Platform - Version 13.4.0.0.0 and laterInformation in this document applies to any platform.
Symptoms
This issue can happen after upgrading the EM to 13.4 or in a new installation of EM 13.4
1. OMS java process is consuming high amount CPU usage intermittently.
2. During this time, users may observe console slowness.
3. $../gc_inst/em/EMGC_OMS1/sysman/log/emctl.log may report the following errors too:
java.net.SocketTimeoutException: Read timed out
4. ReservedCodeCache will be nearly 100% full. This can be verified from:
- EM Console > Navigate to Oracle Weblogic Server target EMGC_OMS*
- Click on Monitoring -> All metrics
- Search for "JVM Memory Pools"
- Expand JVM Memory Pools and select Memory Pool - Current usage
- Select "Code Cache" from the result page and this will load a graph
- This will be for last 24 hours. Verify the usage and see if it is closer to 100%
- Change "View Data" at the top of the table from 24 Hours to 7 days
- Verify the usage and see if it is closer to 100%
5. From the thread dumps which are collected at the time of issue, JVM compiler threads are causing the high CPU.
This can be verified from the following steps:
5.1) Identify the OMS PID using following command
$ps -ef | grep EMGC_OMS1
$MW_HOME/oracle_common/jdk/bin/jstack <OMS PID> > /tmp/oms1_jstack.txt
5.2) Collect CPU usage of OMS threads
$ps -C java -L -o pcpu,cpu,nice,state,cputime,pid,tid | grep <OMS_PID> > /tmp/oms1_thred_cpu.txt
5.3) Convert hexadecimal nid to decimal
Search for "Compiler" key word to locate compiler threads in /tmp/oms1_jstack.txt
There will be multiple compiler threads.
Convert it's nid value from hexadecimal to decimal.
5.4) Find the cpu utilisation of compiler threads
Search for decimal nid value in /tmp/oms1_thred_cpu.txt
First column shows the CPU utilisation
Repeat the above 2 steps for remaining compiler nids
Cause
To view full details, sign in with your My Oracle Support account. |
|
Don't have a My Oracle Support account? Click to get started! |
In this Document
Symptoms |
Cause |
Solution |
References |