My Oracle Support Banner

EM 12c: OMS Crashes with "java.lang.OutOfMemoryError: Java heap space" When Cloning an Agent Using EMCLI (Doc ID 2158820.1)

Last updated on AUGUST 08, 2016

Applies to:

Enterprise Manager Base Platform - Version to [Release 12.1]
Information in this document applies to any platform.


Existing Agent was being cloned onto a new server using "emcli submit_add_host" command, when it was noticed that the OMS had gone down.

$ emcli login -username=sysman
Enter password :
Login successful

$ emcli sync
Synchronized successfully
$ export src_agent=srcagent.domain:3872
$ export session=add_host_emcli_'date +%Y%m%d%H%M'
$ export new_host=tgtagent.domain
$ date ; time emcli submit_add_host -wait_for_completion -platform=226 -installation_base_directory=/u01/app/oracle/product/12.1.0/agent12c 
-credential_name=$creds -deployment_type=CLONE -host_names="${new_host}" -source_agent="$src_agent" -session_name="$session" -additional_parameters="b_startAgent=false"
Mon Jun 27 14:46:56 AEST 2016
Error: Session expired. Run emcli login to establish a session.
real 23m52.19s
user 0m1.71s
sys 0m0.36s

 After 23 minutes, the emcli session had expired as it lost connectivity to the OMS.

-  The <gc_inst>/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs/EMGC_OMS1.out shows:

<Jun 27, 2016 2:58:40 PM AEST> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '14' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "615" seconds working on the request "Workmanager: default, Version: 0, Scheduled=true, Started=true, Started time: 615746 ms
POST /em/console/cli HTTP/1.1
TE: trailers, deflate, gzip, compress
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv: Gecko/20071025 Firefox/ EMCLI Java Browser Fwk_Token:
X-ORCL-EMCLI-INSTALL_LOC: /u01/app/oracle/product/
Accept-Language: en-US
verb_jars_dir: /u01/app/oracle/product/
]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
java.lang.Thread.sleep(Native Method)
sun.reflect.GeneratedMethodAccessor1865.invoke(Unknown Source)
<Jun 27, 2016 3:10:30 PM AEST> <Error> <Kernel> <BEA-000802> <ExecuteRequest failed
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space

-  The <gc_inst>/em/EMGC_OMS1/sysman/log/emctl.msg shows that the OMS had also been re-started by the HealthMonitor thread due to

HealthMonitor Jun 27, 2016 3:10:30 PM OMS Heartbeat Recorder: error: Zero rows updated for this oms during heartbeat
Critical error err=3 detected in module OMS Heartbeat Recorder:
OMS will be restarted. A full thread dump will be generated in the log file
to help Oracle Support analyse the problem.



To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.