EM 12c: OMS Crashes with "java.lang.OutOfMemoryError: Java heap space" When Cloning an Agent Using EMCLI

(Doc ID 2158820.1)

Last updated on AUGUST 08, 2016

Applies to:

Enterprise Manager Base Platform - Version to [Release 12.1]
Information in this document applies to any platform.


Existing Agent was being cloned onto a new server using "emcli submit_add_host" command, when it was noticed that the OMS had gone down.

$ emcli login -username=sysman
Enter password :
Login successful

$ emcli sync
Synchronized successfully
$ export src_agent=srcagent.domain:3872
$ export session=add_host_emcli_'date +%Y%m%d%H%M'
$ export new_host=tgtagent.domain
$ date ; time emcli submit_add_host -wait_for_completion -platform=226 -installation_base_directory=/u01/app/oracle/product/12.1.0/agent12c 
-credential_name=$creds -deployment_type=CLONE -host_names="${new_host}" -source_agent="$src_agent" -session_name="$session" -additional_parameters="b_startAgent=false"
Mon Jun 27 14:46:56 AEST 2016
Error: Session expired. Run emcli login to establish a session.
real 23m52.19s
user 0m1.71s
sys 0m0.36s

 After 23 minutes, the emcli session had expired as it lost connectivity to the OMS.

-  The <gc_inst>/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs/EMGC_OMS1.out shows:

<Jun 27, 2016 2:58:40 PM AEST> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '14' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "615" seconds working on the request "Workmanager: default, Version: 0, Scheduled=true, Started=true, Started time: 615746 ms
POST /em/console/cli HTTP/1.1
TE: trailers, deflate, gzip, compress
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv: Gecko/20071025 Firefox/ EMCLI Java Browser Fwk_Token:
X-ORCL-EMCLI-INSTALL_LOC: /u01/app/oracle/product/
Accept-Language: en-US
verb_jars_dir: /u01/app/oracle/product/
]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
java.lang.Thread.sleep(Native Method)
sun.reflect.GeneratedMethodAccessor1865.invoke(Unknown Source)
<Jun 27, 2016 3:10:30 PM AEST> <Error> <Kernel> <BEA-000802> <ExecuteRequest failed
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space

-  The <gc_inst>/em/EMGC_OMS1/sysman/log/emctl.msg shows that the OMS had also been re-started by the HealthMonitor thread due to

HealthMonitor Jun 27, 2016 3:10:30 PM OMS Heartbeat Recorder: error: Zero rows updated for this oms during heartbeat
Critical error err=3 detected in module OMS Heartbeat Recorder:
OMS will be restarted. A full thread dump will be generated in the log file
to help Oracle Support analyse the problem.



Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms