EM 12c: OMS Crashes with "java.lang.OutOfMemoryError: Java heap space" When Cloning an Agent Using EMCLI
(Doc ID 2158820.1)
Last updated on SEPTEMBER 27, 2019
Applies to:
Enterprise Manager Base Platform - Version 12.1.0.1.0 to 12.1.0.5.0 [Release 12.1]Information in this document applies to any platform.
Symptoms
Existing Agent was being cloned onto a new server using "emcli submit_add_host" command, when it was noticed that the OMS had gone down.
$ emcli login -username=sysman Enter password :
Login successful $ emcli sync Synchronized successfully
$ export src_agent=srcagent.domain:3872
$ export session=add_host_emcli_'date +%Y%m%d%H%M'
$ export new_host=tgtagent.domain
$ date ; time emcli submit_add_host -wait_for_completion -platform=226 -installation_base_directory=/u01/app/oracle/product/12.1.0/agent12c
-credential_name=$creds -deployment_type=CLONE -host_names="${new_host}" -source_agent="$src_agent" -session_name="$session" -additional_parameters="b_startAgent=false"
Mon Jun 27 14:46:56 AEST 2016
Error: Session expired. Run emcli login to establish a session.
real 23m52.19s user 0m1.71s sys 0m0.36s
After 23 minutes, the emcli session had expired as it lost connectivity to the OMS.
- The <gc_inst>/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs/EMGC_OMS1.out shows:
<Jun 27, 2016 2:58:40 PM AEST> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '14' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "615" seconds working on the request "Workmanager: default, Version: 0, Scheduled=true, Started=true, Started time: 615746 ms [ POST /em/console/cli HTTP/1.1 TE: trailers, deflate, gzip, compress User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.9) Gecko/20071025 Firefox/2.0.0.9 EMCLI Java Browser Fwk_Token:12.1.0.5.0: X-ORCL-EMCLI-CLIENT_HOSTNAME: omshost.domain X-ORCL-EMCLI-INSTALL_LOC: /u01/app/oracle/product/12.1.0.5/Middleware/oms/bin X-ORCL-EMCLI-INSTANCE_HOME: /home/oracle Accept-Language: en-US X-ORCL-EMCLI-INSTALL_TYPE: Advanced X-ORCL-EMCLI-LAUNCH_MODE: STANDALONE X-ORCL-EMCLI-CLIENT_VER: 12.1.0.5.0 X-ORCL-EMCLI-NOREGISTER_SET: false verb_jars_dir: /u01/app/oracle/product/12.1.0.5/Middleware/oms/bin/bindings/12.1.0.5.0/.emcli ............
]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace: java.lang.Thread.sleep(Native Method) oracle.sysman.core.agentpush.emcli.AddHostAbstractVerb.printAddHostStatus(AddHostAbstractVerb.java:250) oracle.sysman.core.agentpush.emcli.SubmitAddHostVerb.execute(SubmitAddHostVerb.java:225) oracle.sysman.emSDK.emCLI.CLIPageHandler.render(CLIPageHandler.java:717) sun.reflect.GeneratedMethodAccessor1865.invoke(Unknown Source) .................... <Jun 27, 2016 3:10:30 PM AEST> <Error> <Kernel> <BEA-000802> <ExecuteRequest failed java.lang.OutOfMemoryError: Java heap space. java.lang.OutOfMemoryError: Java heap space
- The <gc_inst>/em/EMGC_OMS1/sysman/log/emctl.msg shows that the OMS had also been re-started by the HealthMonitor thread due to
HealthMonitor Jun 27, 2016 3:10:30 PM OMS Heartbeat Recorder: error: Zero rows updated for this oms during heartbeat Critical error err=3 detected in module OMS Heartbeat Recorder: OMS will be restarted. A full thread dump will be generated in the log file /u01/app/oracle/product/12.1.0.5/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs/EMGC_OMS1.out to help Oracle Support analyse the problem.
Cause
To view full details, sign in with your My Oracle Support account. |
|
Don't have a My Oracle Support account? Click to get started! |
In this Document
| Symptoms |
| Cause |
| Solution |
| References |