EM 13.4: OMS Java Process Consuming High CPU and Causing Console Slowness / OMS Hung Intermittently
(Doc ID 2702019.1)
Last updated on SEPTEMBER 13, 2023
Applies to:
Enterprise Manager Base Platform - Version 13.4.0.0.0 and laterInformation in this document applies to any platform.
Symptoms
This issue can happen after upgrading the EM to 13.4 or in a new installation of EM 13.4
1. OMS java process is consuming high amount CPU usage intermittently.
2. During this time, users may observe console slowness.
3. $../gc_inst/em/EMGC_OMS1/sysman/log/emctl.log may report the following errors too:
java.net.SocketTimeoutException: Read timed out
OR / AND
$../gc_inst/user-projects/domains/GCDomain/servers/EMGC_OMS1/logs/EMGC_OMS1*.out* file may report following errors:
Java HotSpot(TM) 64-Bit Server VM warning: Try increasing the code cache size using -XX:ReservedCodeCacheSize=
CodeCache: size=102400Kb used=101070Kb max_used=101207Kb free=1329Kb
bounds [0x00007f22c1c00000, 0x00007f22c8000000, 0x00007f22c8000000]
total_blobs=19338 nmethods=17539 adapters=1700
compilation: disabled (not enough contiguous free space left)
4. ReservedCodeCache will be nearly 100% full. This can be verified from:
- EM Console > Navigate to Oracle Weblogic Server target EMGC_OMS*
- Click on Monitoring -> All metrics
- Search for "JVM Memory Pools"
- Expand JVM Memory Pools and select Memory Pool - Current usage
- Select "Code Cache" from the result page and this will load a graph
- This will be for last 24 hours. Verify the usage and see if it is closer to 100%
- Change "View Data" at the top of the table from 24 Hours to 7 days
- Verify the usage and see if it is closer to 100
5. From the thread dumps which are collected at the time of issue, JVM compiler threads are causing the high CPU.
This can be verified from the following steps:
5.1) Identify the OMS PID using following command
$ps -ef | grep EMGC_OMS1
$MW_HOME/oracle_common/jdk/bin/jstack <OMS PID> > /tmp/oms1_jstack.txt
5.2) Collect CPU usage of OMS threads
$ps -C java -L -o pcpu,cpu,nice,state,cputime,pid,tid | grep <OMS_PID> > /tmp/oms1_thred_cpu.txt
5.3) Convert hexadecimal nid to decimal
Search for "Compiler" key word to locate compiler threads in /tmp/oms1_jstack.txt
There will be multiple compiler threads.
Convert it's nid value from hexadecimal to decimal.
5.4) Find the cpu utilisation of compiler threads
Search for decimal nid value in /tmp/oms1_thred_cpu.txt
First column shows the CPU utilisation
Repeat the above 2 steps for remaining compiler nids
Cause
To view full details, sign in with your My Oracle Support account. |
|
Don't have a My Oracle Support account? Click to get started! |
In this Document
| Symptoms |
| Cause |
| Solution |
| FAQs |
| References |