EM 12c : emctl start agent Fails With Error 'Fatal agent error: Request Dispatcher failed at Startup' 'EMAgent is Thrashing. Exiting watchdog' 'Ping Manager failed at Startup' (Doc ID 1611173.1)

Last updated on MAY 20, 2016

Applies to:

Enterprise Manager Base Platform - Version 12.1.0.1.0 and later
Information in this document applies to any platform.

Symptoms

$/AGENT_INST/bin/emctl start agent

fails with the following error

Oracle Enterprise Manager 12c Cloud Control 12.1.0.1.0
Copyright (c) 1996, 2011 Oracle Corporation. All rights reserved.
Starting agent ............ failed.
EMAgent is Thrashing. Exiting watchdog
Starting agent ............... failed.
Fatal agent error: Request Dispatcher failed at Startup

/AGENT_INST/sysman/log/gcagent.log reports the following message

2013-12-30 05:55:42,175 [795969:GC.Executor.160646 (host:<FQN>:CCCWatchdog) (host:<FQN>:CCCWatchdog:CCCWatchdog)] ERROR - host:<FQN>:CCCWatchdog:CCCWatchdog
java.lang.OutOfMemoryError: unable to create new native thread


$export EMCTL_DEBUG=true

$/AGENT_INST/bin/emctl start agent

fails with following error

./emctl: line 138: /tmp/emctl.env.25096: Read-only file system
emctl: launching /u01/app/oracle/product/12.1.0/agent12c/core/12.1.0.2.0/perl/bin/perl /u01/app/oracle/product/12.1.0/agent12c/core/12.1.0.2.0/bin/emctl.pl start agent
./emctl: line 443: /tmp/emctl.env.25096.2: Read-only file system
./emctl: line 444: /tmp/emctl.env.25096.diff: Read-only file system

OR

$/AGENT_INST/bin/emctl start agent

failes with following error

Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.
Starting agent ........ failed.
Ping Manager failed at Startup
Consult emctl.log and emagent.nohup in: /data/oracle/agent/product/agent12c/agent_inst/sysman/log

/AGENT_INST/sysman/log/gcagent.log reports the following message

Caused by: oracle.sysman.gcagent.ping.PingTimestampFileInitException: Failed
to initialize agntstmp.txt file
at oracle.sysman.gcagent.ping.PingAgentTimestampFile.init(PingAgentTimestampFile.java:164)
at oracle.sysman.gcagent.ping.PingManager.initTimestampFile(PingManager.java:589)
at oracle.sysman.gcagent.ping.PingComponent.tmNotifier(PingComponent.java:74)
at oracle.sysman.gcagent.tmmain.lifecycle.TMComponentSvc.invokeNotifier(TMComponentSvc.java:897)
at oracle.sysman.gcagent.tmmain.lifecycle.TMComponentSvc.invokeInitializationStep(TMComponentSvc.java:980)
... 5 more
Caused by: java.io.IOException: Can't remove existing backup file
[/oem/app/oracle/product/12c/agent_inst/sysman/emd/agntstmp.txt.bak]
at oracle.sysman.gcagent.common.FileUtils.safeSetContents(FileUtils.java:245)

OR

$/AGENT_INST/bin/emctl start agent

fails with following error

Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
Starting agent ........ failed.
Ping Manager failed at Startup
Consult emctl.log and emagent.nohup in: /data/oracle/agent/product/agent12c/agent_inst/sysman/log

/AGENT_INST/sysman/log/gcagent.log reports file IO exception

2014-11-07 21:22:44,286 [1930450:GC.Executor.778231 (oracle_database:sgpssp01_prd:log_full) (oracle_database:sgpssp01_prd:log_full:dumpFull)] ERROR - File not found exception
java.io.FileNotFoundException: /data/oracle/agent/product/agent12c/agent_inst/sysman/emd/state/statemgmt/oracle_database/sgpssp01_prd/SevDiff/1.tmp (Invalid file system control data detected.)

Deleting file  has filesystem related error

$ rm agntstmp.txt

rm: agntstmp.txt not removed.
Invalid file system control data detected

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms