EM12c, EM13c: Enterprise Manager Cloud Control Heap Usage Percent Metric for EMGC_OMSn target Reports Frequent Warning or Critical Alerts, Continuous Growth (Doc ID 2123712.1)

Last updated on SEPTEMBER 10, 2020

Applies to:

Symptoms

Frequent Warning or Critical Alerts and / or Email Notifications are generated from Enterprise Manager (EM) Cloud Control for Heap Usage % metric for EMGC_OMSn target. Eg:

 Target type=Oracle WebLogic Server
 Target name=/EMGC_GCDomain/GCDomain/EMGC_OMS1
 Categories=Capacity
 Message=The heap usage is 96%.
 Severity=Critical
 Event reported time=Jul 29, 2014 11:15:05 PM EDT
 Associated Incident Id=140
 Associated Incident Status=New
 Associated Incident Owner=
 Associated Incident Acknowledged By Owner=No
 Associated Incident Priority=None
 Associated Incident Escalation Level=0
 Event Type=Metric Alert
 Event name=jvm:heapUsedPercentage.value
 Metric Group=JVM Metrics
 Metric=Heap Usage (%)
 Metric value=96

Navigating to Cloud Control Console -> Targets -> All targets -> EMGC_OMSn target (eg: /EMGC_GCDomain/GCDomain/EMGC_OMS1) -> Weblogic Server -> Monitoring -> All Metrics -> JVM Metrics -> Heap Usage (%)
it can be observed that the Utilization percent grows continuously every day.

Additionally, the following symptoms may be observed:

1.    Frequent log rotation for access.log files reported in <gc_inst>/user_projects/domains/GCDomain/servers/EMGC_OMSn/logs/EMGC_OMSn.out via entries such as:

<Jul 29, 2014 11:21:31 PM WEST> <Notice> <LoggingService> <BEA-320400> <The log file /app/oracle/oem/gc_inst/user_projects
/domains/GCDomain/servers/EMGC_OMS1/logs/access.log will be rotated. Reopen the log file if tailing has stopped. This can happen on some platforms like Windows.>

<Jul 29, 2014 11:21:31 PM WEST> <Notice> <LoggingService> <BEA-320401> <The log file has been rotated to /app/oracle/oem/gc_inst/user_projects
/domains/GCDomain/servers/EMGC_OMS1/logs/access.log01256. Log messages will continue to be logged in /app/oracle/oem/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs/access.log.>

2. Errors such as "java.lang.OutOfMemoryError: Java heap space" and "danger of OutOfMemoryError" reported in <gc_inst>/user_projects/domains/GCDomain/servers/EMGC_OMSn/logs/EMGC_OMSn.out

<Jul 29, 2014 11:23:10 PM WEST> <Critical> <Health> <BEA-310003> <Free memory in the server is 3,416,688 bytes. 
There is danger of OutOfMemoryError>
<Jul 29, 2014 11:47:40 PM WEST> <Error> <Server> <BEA-002608> <The ListenThread because of an error: java.lang.OutOfMemoryError: 
Java heap space java.lang.OutOfMemoryError: Java heap space

3. High number (about 30 times per min) of occurrences for GET requests for a particular page by an EM Administrator reported in <gc_inst>/user_projects/domains/GCDomain/servers/EMGC_OMSn/logs/access.log*

<IP ADD> - OPERADOR [29/Jul/2014:11:01:35 +0000] "GET /em/faces/core-groups-systemHome?target=ORCL*

In a time period of around 3 hours, there could be as high as 2957 GET calls from the EM Administrator user.

Changes

 There is an user session via browser established to the EM 12c console for the particular target which has Auto-Refresh turned on for continuous monitoring.

  OR

 There is a Browser Plugin/Add-on configured to access Cloud Control Console page for a particular target for continuous monitoring (eg: plugin iMacros Sidebar)

Cause

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.