EM 12c, EM 13c: Enterprise Manager Cloud Control Agent Start and Status Commands Work but Stop and Upload, Failing with Error: Agent is not Running (Doc ID 1901555.1)

Last updated on SEPTEMBER 09, 2024

Applies to:

Symptoms

The 12c Agent can startup fine and 'emctl status agent' works:

./emctl status agent
Oracle Enterprise Manager Cloud Control 12c Release 3  
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version     : 12.1.0.3.0
OMS Version       : 12.1.0.3.0
Protocol Version  : 12.1.0.1.0
Agent Home        : /oracle/agent12c/agent_inst
Agent Binaries    : /oracle/agent12c/core/12.1.0.3.0
Agent Process ID  : 12910596
Parent Process ID : 9044098
Agent URL         : https://<Agent host name>.<domain>:3872/emd/main/
Repository URL    : https://<OMS host name>.<domain>:4903/empbs/upload
Started at        : 2014-05-15 12:30:59
Started by user   : <username>
Last Reload       : (none)
Last successful upload                       : 2014-05-22 14:17:34
Last attempted upload                        : 2014-05-22 14:17:34
Total Megabytes of XML files uploaded so far : 9.11
Number of XML files pending upload           : 0
Size of XML files pending upload(MB)         : 0
Available disk space on upload filesystem    : 45.39%
Collection Status                            : Collections enabled
Heartbeat Status                             : Ok
Last attempted heartbeat to OMS              : 2014-05-22 14:25:13
Last successful heartbeat to OMS             : 2014-05-22 14:25:13
Next scheduled heartbeat to OMS              : 2014-05-22 14:26:13

---------------------------------------------------------------
Agent is Running and Ready

The targets are discovered in the console and their status is also shown correctly in the Console UI.
But trying to stop the agent fails with:

cd /oracle/agent12c/core/12.1.0.3.0/bin
./emctl stop agent
Oracle Enterprise Manager Cloud Control 12c Release 3  
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent is Not Running

-  'emctl upload' and 'emctl pingOMS' also fail with the same error but clicking on the 'Upload Metric Data' button in the agent homepage from console works fine and does not report any error:

 ./emctl upload
Oracle Enterprise Manager Cloud Control 12c Release 3  
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent is Not Running

- For the upload action, the <AGENT_INST>/sysman/log/emctl.log shows:

13304030 :: Thu May 22 13:34:14 2014::Initializing the agent command locking system
13304030 :: Thu May 22 13:34:14 2014::AgentStatus.pm:Processing upload
13304030 :: Thu May 22 13:34:15 2014::AgentStatus.pm:emdctl status agent returned  1
13304030 :: Thu May 22 13:34:15 2014::Cleaning up agent command lock
13304030 :: Thu May 22 13:34:15 2014::AgentCommandLock:closed file handle of emctl lockfile

 -  The agent process is running at the OS:

ps -ef | grep agent
oracle  9044098        1   0   May 15      -  0:21 /oracle/appgrid/agent12c/core/12.1.0.3.0/perl/bin/perl /oracle/appgrid/agent12c/core/12.1.0.3.0/bin/emwd.pl agent /oracle/appgrid/agent12c/agent_inst/sysman/log/emagent.nohup

oracle 12910596  9044098   0   May 15      - 14:52 /oracle/appgrid/agent12c/core/12.1.0.3.0/jdk/bin/java -Xmx160M -server -Djava.security.egd=file:///dev/./urandom -Dsun.lang.ClassLoader.allowArraySyntax=true -Xgcpolicy:gencon -Dwatchdog.pid=9044098 -cp /oracle/appgrid/agent12c/core/12.1.0.3.0/jdbc/lib/ojdbc5.jar:/oracle/appgrid/agent12c/core/12.1.0.3.0/ucp/lib/ucp.jar:/oracle/appgrid/agent12c/core/12.1.0.3.0/modules/oracle.http_client_11.1.1.jar:/oracle/appgrid/agent12c/core/12.1.0.3.0/lib/xmlparserv2.jar:/oracle/appgrid/agent12c/core/12.1.0.3.0/lib/jsch.jar:/oracle/appgrid/agent12c/core/12.1.0.3.0/lib/optic.jar:/oracle/appgrid/agent12c/core/12.1.0.3.0/modules/oracle.dms_11.1.1/dms.jar:/oracle/appgrid/agent12c/core/12.1.0.3.0/modules/oracle.odl_11.1.1/ojdl.jar:/oracle/appgrid/agent12c/core/12.1.0.3.0/modules/oracle.odl_11.1.1/ojdl2.jar:/oracle/appgrid/agent12c/core/12.1.0.3.0/sysman/jlib/log4j-core.jar:/oracle/appgrid/agent12c/core/12.1.0.3.0/jlib/gcagent_core.jar:/oracle/appgrid/agent12c/core/12.1.0.3.0/sysman/jlib/emagentSDK-intg.jar:/oracle/appgrid/agent12c/core/12.1.0.3.0/sysman/jlib/emagentSDK.jar oracle.sysman.gcagent.tmmain.TMMain

Cause

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.