Agent 12c : AgentDeploy.sh emctl start agent Fails with 'java.lang.OutOfMemoryError: Java heap space .. java.io.UnixFileSystem.list(Native Method)' (Doc ID 2257211.1)

Last updated on MAY 11, 2017

Applies to:

Enterprise Manager Base Platform - Version 10.2.0.4 to 12.1.0.5.0 [Release 10.2 to 12.1]
Information in this document applies to any platform.

Symptoms

ISSUE 1 : agentDeploy.sh fails @ plugins deploy phase resulted OOME

Agent Deployment using agentDeploy.sh fails @ plugins deploy phase resulted OOME (Out Of Memory Excption)

From AGENT_HOME/cfgtoollogs/agentDeploy/agentDeploy_2017-03-21_09-09-00-AM.log

INFO: Plugin homes:
INFO:/opt/mis/oracle/product/new_agent12c/core/12.1.0.4.0/oui/bin/runConfig.sh ORACLE_HOME=/opt/mis/oracle/product/new_agent12c/core/12.1.0.4.0 RESPONSE_FILE=/opt/mis/oracle/product/new_agent12c/core/12.1.0.4.0/agent.rsp
ACTION=configure MODE=perform COMPONENT_XML={oracle.sysman.top.agent.11_1_0_1_0.xml} RERUN=true completed
with status=3
SEVERE:
ERROR: Agent Configuration Failed
SEVERE:
ERROR: Agent Deployment Failed
From agentplugindeploy_2017_03_21_09_10_18.log.err
Exception in thread "main-14814" java.lang.OutOfMemoryError: Java heapspace
at java.io.UnixFileSystem.list(Native Method)
at java.io.File.list(File.java:973)
at javax.crypto.SunJCE_b.j(DashoA13*..)
at javax.crypto.SunJCE_b.d(DashoA13*..)
at javax.crypto.SunJCE_b.c(DashoA13*..)
at javax.crypto.SunJCE_b$1.run(DashoA13*..)
at java.security.AccessController.doPrivileged(Native Method)
at javax.crypto.SunJCE_b.<clinit>(DashoA13*..)
at javax.crypto.Mac.getInstance(DashoA13*..)
at oracle.security.crypto.cert.PKCS12.verify(PKCS12.java:406)
at oracle.security.pki.OracleKeyStoreSpi.engineLoad(UnknownSource)
at java.security.KeyStore.load(KeyStore.java:1183)
at oracle.sysman.gcagent.comm.http.AgentServerWallet.<init>(AgentServerWallet.java:109)
at oracle.sysman.gcagent.comm.agent.http.SSLInit.getJSSESupportedSuites(SSLInit.java:188)
at oracle.sysman.gcagent.comm.agent.http.SSLInit.computeCipherSuites(SSLInit.java:145)
at oracle.sysman.gcagent.comm.agent.http.SSLInit.configure(SSLInit.java:93)
at oracle.sysman.gcagent.comm.http.LocalSecurityContext.<init>(LocalSecurityContxt.java:55)
at oracle.sysman.gcagent.clients.emdctlj.EmdCtl.main(EmdCtl.java:394)

 

ISSUE 2 : emctl start agent fails with OOME

Executing manual clone of agent image and start agent fails with OOME

$./emctl start agent

Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
Starting agent ...........................................................................................................................failed.
Consult emctl.log and emagent.nohup in: /opt/mis/oracle/product/agent12c/agent_inst/sysman/log

$./emctl clearstate agent

Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
Exception in thread "main-8436" java.lang.OutOfMemoryError: Java heap space
at java.io.UnixFileSystem.list(Native Method)
at java.io.File.list(File.java:973)
at javax.crypto.SunJCE_b.j(DashoA13*..)
at javax.crypto.SunJCE_b.d(DashoA13*..)
at javax.crypto.SunJCE_b.c(DashoA13*..)
at javax.crypto.SunJCE_b$1.run(DashoA13*..)
at java.security.AccessController.doPrivileged(Native Method)
at javax.crypto.SunJCE_b.<clinit>(DashoA13*..)
at javax.crypto.Mac.getInstance(DashoA13*..)
at oracle.security.crypto.cert.PKCS12.verify(PKCS12.java:406)
at oracle.security.pki.OracleKeyStoreSpi.engineLoad(Unknown Source)
at java.security.KeyStore.load(KeyStore.java:1185)
at oracle.sysman.gcagent.comm.http.AgentServerWallet.<init>(AgentServerWallet.java:109)
at oracle.sysman.gcagent.comm.agent.http.SSLInit.getJSSESupportedSuites(SSLInit.java:188)
at oracle.sysman.gcagent.comm.agent.http.SSLInit.computeCipherSuites(SSLInit.java:145)
at oracle.sysman.gcagent.comm.agent.http.SSLInit.configure(SSLInit.java:93)
at oracle.sysman.gcagent.comm.http.LocalSecurityContext.<init>(LocalSecurityContext.java:55)
at oracle.sysman.gcagent.clients.emdctlj.EmdCtl.main(EmdCtl.java:394)



Verified working and problematic node's /var/tmp details.

Problematic node :

ebstst2db1-<oracle>-1% cd /var/tmp
ebstst2db1-<oracle>-2% ls -1 | wc -l
51 high67183  <<< number of file  are high @ tmp location

Working node :

ebsuat1db1-<oracle>-1% cd /var/tmp
ebsuat1db1-<oracle>-2% ls -1 | wc -l
10299

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms