EM 12c: Enterprise Manager 12.1.0.2 Cloud Control Agent is not Restarted Automatically by the Watchdog if it Crashes with Message: java.lang.OutOfMemoryError: Java heap space (Doc ID 1544013.1)

Last updated on JULY 28, 2017

Applies to:

Enterprise Manager Base Platform - Version 12.1.0.1.0 to 12.1.0.2.0 [Release 12.1]
Information in this document applies to any platform.

Symptoms

The Enterprise Manager (EM) 12c Cloud Control Agent crashes due to 'java.lang.OutOfMemoryError: Java heap space'  but was not automatically restarted by the watchdog process.

The <AGENT_INST>/sysman/log/gcagent.log file shows:

2013-02-21 16:43:43,776 [1:main] ERROR - agent main throw an error
java.lang.OutOfMemoryError: Java heap space
....
2013-02-21 16:44:05,495 [347114:GC.Executor.137521 (host:agentmachine.domain:FilesystemsLinux) (host:agentmachine.domain:FilesystemsLinux:Filesystems)] ERROR - host:agentmachine.domain:FilesystemsLinux:Filesystems
java.lang.OutOfMemoryError: Java heap space
2013-02-21 16:44:05,498 [91:3C0B0719:HTTP Listener-91] WARN - *jetty*: handle failed
java.lang.IllegalStateException: Internal error
    at com.sun.net.ssl.internal.ssl.SSLEngineImpl.initHandshaker(SSLEngineImpl.java:414)
    at com.sun.net.ssl.internal.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:959)
    at com.sun.net.ssl.internal.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:837)
    at com.sun.net.ssl.internal.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:713)
    at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:607)
    at org.eclipse.jetty.http.ssl.SslSelectChannelEndPoint.unwrap(SslSelectChannelEndPoint.java:666)
    at org.eclipse.jetty.http.ssl.SslSelectChannelEndPoint.fill(SslSelectChannelEndPoint.java:335)
    at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:281)
    at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:203)
    at org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:406)
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:462)
    at oracle.sysman.gcagent.comm.agent.http.HTTPListenerThreadPool$JobSleeve.run(HTTPListenerThreadPool.java:256)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:436)
    at oracle.sysman.gcagent.util.system.GCAThread$RunnableWrapper.run(GCAThread.java:141)
    at java.lang.Thread.run(Thread.java:662)
2013-02-21 16:44:05,497 [347235:F717DB89:GC.SysExecutor.16853 (Ping OMS)] INFO - attempting another heartbeat
2013-02-21 16:44:05,504 [32:F9C26A76] INFO - *jetty*: Graceful shutdown ContextHandler@6e0f0ad@6e0f0ad/emd/lifecycle/main,null
2013-02-21 16:44:05,504 [32:F9C26A76] INFO - *jetty*: Graceful shutdown HTTPLifecycleHandler@19d8f162
2013-02-21 16:44:05,504 [32:F9C26A76] INFO - *jetty*: Graceful shutdown ContextHandler@71cf662f@71cf662f/emd/main,null
2013-02-21 16:44:05,504 [32:F9C26A76] INFO - *jetty*: Graceful shutdown HTTPRequestHandler@673dd208
2013-02-21 16:44:05,504 [32:F9C26A76] INFO - *jetty*: Graceful shutdown ServletContextHandler@8135daf@8135daf/emd/browser,null
2013-02-21 16:44:05,504 [32:F9C26A76] INFO - *jetty*: Graceful shutdown ContextHandler@ba8402c@ba8402c/,null
2013-02-21 16:44:09,843 [347099:A0157156] INFO - Agent exiting with exit code 57
2013-02-21 16:44:09,891 [32:F9C26A76] INFO - *jetty*: Shutdown hook complete

 

The <AGENT_INST>/sysman/log/emagent.nohup file shows:

----- Thu Feb 21 16:44:05 2013::8589::Checking status of EMAgent : 8659 -----
Agent is going down due to an OutOfMemoryError
----- Thu Feb 21 16:44:14 2013::8589::Checking status of EMAgent : 8659 -----
----- Thu Feb 21 16:44:16 2013::8589::EMAgent exited at Thu Feb 21 16:44:16 2013 with return value 55. -----
----- Thu Feb 21 16:44:16 2013::8589::EMAgent has exited due to initialization failure. -----
----- Thu Feb 21 16:44:16 2013::8589::Stopping other components. -----
----- Thu Feb 21 16:44:16 2013::8589::Commiting Process death. -----
----- Thu Feb 21 16:44:16 2013::8589::Generating additional diagnostics for 8659 -----
----- Thu Feb 21 16:44:16 2013::8589::Generating openfiles report in /oem/app/oracle/product/12c/agent_inst/sysman/log/lsof_8659 -----
----- Thu Feb 21 16:44:16 2013::8589::Generating netstat report in /oem/app/oracle/product/12c/agent_inst/sysman/log/netstat_8659 -----
----- Thu Feb 21 16:44:16 2013::8589::Generating all host processes report in /oem/app/oracle/product/12c/agent_inst/sysman/log/allprocesses_8659 -----
----- Thu Feb 21 16:44:16 2013::8589::Exiting watchdog loop
 -----
 --- EMState agent

 

The Agent must be started manually using:

cd <AGENT_HOME>
emctl start agent

 

Changes

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms