12c EM: Agent Start / Status Commands Work but Stop / Upload Fails with 'Agent is not Running' (Doc ID 1901555.1)

Last updated on APRIL 27, 2017

Applies to:

Enterprise Manager Base Platform - Version 12.1.0.3.0 and later
IBM AIX on POWER Systems (64-bit)

Symptoms

The 12c Agent can startup fine and 'emctl status agent' works:

./emctl status agent
Oracle Enterprise Manager Cloud Control 12c Release 3  
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version     : 12.1.0.3.0
OMS Version       : 12.1.0.3.0
Protocol Version  : 12.1.0.1.0
Agent Home        : /oracle/agent12c/agent_inst
Agent Binaries    : /oracle/agent12c/core/12.1.0.3.0
Agent Process ID  : 12910596
Parent Process ID : 9044098
Agent URL         : https://agentmachine.domain:3872/emd/main/
Repository URL    : https://omsmachine.domain:4903/empbs/upload
Started at        : 2014-05-15 12:30:59
Started by user   : oracle
Last Reload       : (none)
Last successful upload                       : 2014-05-22 14:17:34
Last attempted upload                        : 2014-05-22 14:17:34
Total Megabytes of XML files uploaded so far : 9.11
Number of XML files pending upload           : 0
Size of XML files pending upload(MB)         : 0
Available disk space on upload filesystem    : 45.39%
Collection Status                            : Collections enabled
Heartbeat Status                             : Ok
Last attempted heartbeat to OMS              : 2014-05-22 14:25:13
Last successful heartbeat to OMS             : 2014-05-22 14:25:13
Next scheduled heartbeat to OMS              : 2014-05-22 14:26:13

---------------------------------------------------------------
Agent is Running and Ready

The targets are discovered in the console and their status is also shown correctly in the Console UI.
But trying to stop the agent fails with:

cd /oracle/agent12c/core/12.1.0.3.0/bin
./emctl stop agent

Oracle Enterprise Manager Cloud Control 12c Release 3  
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent is Not Running

'emctl upload' and 'emctl pingOMS' also fail with the same error but clicking on the 'Upload Metric Data' button in the agent homepage from console works fine and does not report any error:

 ./emctl upload
Oracle Enterprise Manager Cloud Control 12c Release 3  
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent is Not Running

- For the upload action, the <AGENT_INST>/sysman/log/emctl.log shows:

13304030 :: Thu May 22 13:34:14 2014::Initializing the agent command locking system
13304030 :: Thu May 22 13:34:14 2014::AgentStatus.pm:Processing upload
13304030 :: Thu May 22 13:34:15 2014::AgentStatus.pm:emdctl status agent returned  1
13304030 :: Thu May 22 13:34:15 2014::Cleaning up agent command lock
13304030 :: Thu May 22 13:34:15 2014::AgentCommandLock:closed file handle of emctl lockfile

 -  The agent process is running at the OS:

ps -ef | grep agent
oracle  9044098        1   0   May 15      -  0:21 /oracle/appgrid/agent12c/core/12.1.0.3.0/perl/bin/perl /oracle/appgrid/agent12c/core/12.1.0.3.0/bin/emwd.pl agent /oracle/appgrid/agent12c/agent_inst/sysman/log/emagent.nohup

oracle 12910596  9044098   0   May 15      - 14:52 /oracle/appgrid/agent12c/core/12.1.0.3.0/jdk/bin/java -Xmx160M -server -Djava.security.egd=file:///dev/./urandom -Dsun.lang.ClassLoader.allowArraySyntax=true -Xgcpolicy:gencon -Dwatchdog.pid=9044098 -cp /oracle/appgrid/agent12c/core/12.1.0.3.0/jdbc/lib/ojdbc5.jar:/oracle/appgrid/agent12c/core/12.1.0.3.0/ucp/lib/ucp.jar:/oracle/appgrid/agent12c/core/12.1.0.3.0/modules/oracle.http_client_11.1.1.jar:/oracle/appgrid/agent12c/core/12.1.0.3.0/lib/xmlparserv2.jar:/oracle/appgrid/agent12c/core/12.1.0.3.0/lib/jsch.jar:/oracle/appgrid/agent12c/core/12.1.0.3.0/lib/optic.jar:/oracle/appgrid/agent12c/core/12.1.0.3.0/modules/oracle.dms_11.1.1/dms.jar:/oracle/appgrid/agent12c/core/12.1.0.3.0/modules/oracle.odl_11.1.1/ojdl.jar:/oracle/appgrid/agent12c/core/12.1.0.3.0/modules/oracle.odl_11.1.1/ojdl2.jar:/oracle/appgrid/agent12c/core/12.1.0.3.0/sysman/jlib/log4j-core.jar:/oracle/appgrid/agent12c/core/12.1.0.3.0/jlib/gcagent_core.jar:/oracle/appgrid/agent12c/core/12.1.0.3.0/sysman/jlib/emagentSDK-intg.jar:/oracle/appgrid/agent12c/core/12.1.0.3.0/sysman/jlib/emagentSDK.jar oracle.sysman.gcagent.tmmain.TMMain


Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms