EM 12c: OMS Crashes with "java.lang.OutOfMemoryError: Java heap space" When Cloning an Agent Using EMCLI (Doc ID 2158820.1)

Last updated on AUGUST 08, 2016

Applies to:

Enterprise Manager Base Platform - Version 12.1.0.1.0 to 12.1.0.5.0 [Release 12.1]
Information in this document applies to any platform.

Symptoms

Existing Agent was being cloned onto a new server using "emcli submit_add_host" command, when it was noticed that the OMS had gone down.

$ emcli login -username=sysman
Enter password :
Login successful

$ emcli sync
Synchronized successfully
$ export src_agent=srcagent.domain:3872
$ export session=add_host_emcli_'date +%Y%m%d%H%M'
$ export new_host=tgtagent.domain
$ date ; time emcli submit_add_host -wait_for_completion -platform=226 -installation_base_directory=/u01/app/oracle/product/12.1.0/agent12c 
-credential_name=$creds -deployment_type=CLONE -host_names="${new_host}" -source_agent="$src_agent" -session_name="$session" -additional_parameters="b_startAgent=false"
Mon Jun 27 14:46:56 AEST 2016
Error: Session expired. Run emcli login to establish a session.
real 23m52.19s
user 0m1.71s
sys 0m0.36s

 After 23 minutes, the emcli session had expired as it lost connectivity to the OMS.

-  The <gc_inst>/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs/EMGC_OMS1.out shows:

<Jun 27, 2016 2:58:40 PM AEST> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '14' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "615" seconds working on the request "Workmanager: default, Version: 0, Scheduled=true, Started=true, Started time: 615746 ms
[
POST /em/console/cli HTTP/1.1
TE: trailers, deflate, gzip, compress
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.9) Gecko/20071025 Firefox/2.0.0.9 EMCLI Java Browser Fwk_Token:12.1.0.5.0:
X-ORCL-EMCLI-CLIENT_HOSTNAME: omshost.domain
X-ORCL-EMCLI-INSTALL_LOC: /u01/app/oracle/product/12.1.0.5/Middleware/oms/bin
X-ORCL-EMCLI-INSTANCE_HOME: /home/oracle
Accept-Language: en-US
X-ORCL-EMCLI-INSTALL_TYPE: Advanced
X-ORCL-EMCLI-LAUNCH_MODE: STANDALONE
X-ORCL-EMCLI-CLIENT_VER: 12.1.0.5.0
X-ORCL-EMCLI-NOREGISTER_SET: false
verb_jars_dir: /u01/app/oracle/product/12.1.0.5/Middleware/oms/bin/bindings/12.1.0.5.0/.emcli
............
]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
java.lang.Thread.sleep(Native Method)
oracle.sysman.core.agentpush.emcli.AddHostAbstractVerb.printAddHostStatus(AddHostAbstractVerb.java:250)
oracle.sysman.core.agentpush.emcli.SubmitAddHostVerb.execute(SubmitAddHostVerb.java:225)
oracle.sysman.emSDK.emCLI.CLIPageHandler.render(CLIPageHandler.java:717)
sun.reflect.GeneratedMethodAccessor1865.invoke(Unknown Source)
....................
<Jun 27, 2016 3:10:30 PM AEST> <Error> <Kernel> <BEA-000802> <ExecuteRequest failed
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space

-  The <gc_inst>/em/EMGC_OMS1/sysman/log/emctl.msg shows that the OMS had also been re-started by the HealthMonitor thread due to

HealthMonitor Jun 27, 2016 3:10:30 PM OMS Heartbeat Recorder: error: Zero rows updated for this oms during heartbeat
Critical error err=3 detected in module OMS Heartbeat Recorder:
OMS will be restarted. A full thread dump will be generated in the log file
/u01/app/oracle/product/12.1.0.5/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs/EMGC_OMS1.out
to help Oracle Support analyse the problem.

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms