EM 12c, EM 13c: Enterprise Manager Cloud Control Agent emctl Commands Fail with Error: EMD upload error:full upload has failed and ORA-20500

Last updated on APRIL 28, 2018

Applies to:

Symptoms

$/AGENT_INST/bin/emctl upload agent

fails with the following error

Oracle Enterprise Manager Cloud Control 12c Release 3  
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
EMD upload error:full upload has failed: uploadXMLFiles skipped :: OMS version not checked yet. If this issue persists check trace files for ping to OMS related errors. (OMS_DOWN)

$/AGENT_INST/bin/emctl status agent

Shows Heartbeat Status 'OMS is unreachable [bad response]'

Oracle Enterprise Manager Cloud Control 12c Release 3  
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version     : 12.1.0.3.0
OMS Version       : (unknown)
Protocol Version  : 12.1.0.1.0
Agent Home        : /oem10g/oracle/agent12c/agent_inst
Agent Binaries    : /oem10g/oracle/agent12c/core/12.1.0.3.0
Agent Process ID  : 10490
Parent Process ID : 1397
Agent URL         : https://current_agentname:1839/emd/main/
Repository URL    : https://current_agentname:4901/empbs/upload
Started at        : 2013-09-23 09:54:48
Started by user   : oem12c
Last Reload       : (none)
Last successful upload                       : (none)
Last attempted upload                        : (none)
Total Megabytes of XML files uploaded so far : 0
Number of XML files pending upload           : 0
Size of XML files pending upload(MB)         : 0
Available disk space on upload filesystem    : 91.31%
Collection Status                            : Collections enabled
Heartbeat Status                             : OMS is unreachable [bad response]
Last attempted heartbeat to OMS              : 2013-09-24 13:11:34
Last successful heartbeat to OMS             : (none)
Next scheduled heartbeat to OMS              : 2013-09-24 13:12:04

----------------------------------------------------------
Agent is Running and Ready

$/AGENT_INST/bin/emctl pingOMS

fails with the following error

Oracle Enterprise Manager Cloud Control 12c Release 3  
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
EMD pingOMS error: OMS sent an invalid response: "ERROR- Failed to update Target type Metadata"

/gc_inst/em/EMGC_OMS1/sysman/log/emoms_pbs.log reports the following ORA error

2013-09-24 13:59:23,535 [[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'] ERROR gcloader.Recei
ver logp.251 - Error occurred in checking information between Agent and OMS
oracle.sysman.emSDK.core.target.DuplicateEntityException: ORA-20500: Multiple agents detected for :(host = current_host_name
)
ORA-06512: at "SYSMAN.EM_TARGET", line 3021
ORA-06512: at "SYSMAN.EM_TARGET", line 3317
ORA-06512: at "SYSMAN.EM_TARGET", line 3992
ORA-06512: at line 1

  at oracle.sysman.emdrep.target.EntityUtil.createNYMEntities(EntityUtil.java:1327)
  at oracle.sysman.emSDK.core.target.EntityManager.createNYMEntities(EntityManager.java:232)
  at oracle.sysman.core.pbs.gcloader.Receiver.doGet(Receiver.java:859)

/AGENT_INST/sysman/log/gcagent.log reports the following error

2013-09-23 19:26:22,489 [1843:CB448347:GC.SysExecutor.670 (AgentSystemMonitorTask)] WARN - Subsystem (Ping Manager) returned bad status of {+ Ping Manager: *Warning* +}
2013-09-23 19:26:22,547 [1845:CFC1012A] INFO - attempting initial heartbeat
2013-09-23 19:26:22,754 [1843:CB448347] WARN - Subsystem (Upload Manager) returned bad status of {+ Upload Manager: *Warning* +}
2013-09-23 19:26:22,966 [1845:CFC1012A] WARN - improper ping interval (EM_PING_NOTIF_RESPONSE: ERROR- Failed to update Target type Metadata)
2013-09-23 19:26:22,993 [1845:CFC1012A] WARN - Ping protocol error
o.s.gcagent.ping.PingProtocolException [OMS sent an invalid response: "ERROR- Failed to update Target type Metadata"]
2013-09-23 19:26:53,123 [1854:EE8796A6:GC.SysExecutor.675 (Ping OMS)] INFO - attempting initial heartbeat
2013-09-23 19:26:53,340 [1854:EE8796A6] WARN - improper ping interval (EM_PING_NOTIF_RESPONSE: ERROR- Failed to update Target type Metadata)
2013-09-23 19:26:53,355 [1854:EE8796A6] WARN - Ping protocol error
o.s.gcagent.ping.PingProtocolException [OMS sent an invalid response: "ERROR- Failed to update Target type Metadata"]

Cause