EM 11g: Enterprise Manager 11.1.0.1 Grid Control Agent Upload Fails With Message: ERROR-400|OMS already processing a Metadata or Severity file from this Agent (Doc ID 1536095.1)

Last updated on AUGUST 30, 2014

Applies to:

Enterprise Manager Base Platform - Version 11.1.0.1 to 11.1.0.1.7 [Release 11.1]
Information in this document applies to any platform.

Symptoms

Agent is uploading files to the OMS in a very slow mode. If the Agent is stopped and re-started, the files get uploaded very fast for a while but eventually it slows down considerably and seems to be unable to keep up.

-  The <AGENT_HOME>/sysman/log/emagent.trc file shows:

2013-02-26 13:54:19,166 Thread-377 WARN upload: FxferSend: received http error in header from repository: http://omsmachine.domain:4889/em/upload/
ERROR-400|OMS already processing a Metadata or Severity file from this Agent. Agent should retry later.Error obtaining upload lock in OMS on emd_url:http://agentmachine.domain:3872/emd/main
2013-02-26 13:54:19,166 Thread-377 DEBUG upload: nmehum_uploadOneXMLFile Closed file A0000091.xml, ret = 0
2013-02-26 13:54:19,166 Thread-377 ERROR upload: Failed to upload file A0000091.xml: HTTP error. Response received: ERROR-400|OMS already processing a Metadata or Severity file from this Agent. Agent should retry later.Error obtaining upload lock in OMS on emd_url:http://agentmachine.domain:3872/emd/main
2013-02-26 13:54:19,166 Thread-377 ERROR upload: 1 Failure(s) in a row or XML error for A0000091.xml, retcode = -2, we give up
2013-02-26 13:54:19,166 Thread-377 DEBUG upload: Done uploading. retcode = -2
2013-02-26 13:54:19,166 Thread-377 DEBUG upload: Error in uploadXMLFiles. Trying again in 103.00 seconds or earlier.
2013-02-26 13:54:19,166 Thread-377 DEBUG upload: Upload completed
...

2013-02-26 15:20:36,541 Thread-830 DEBUG upload: Upload already running, aborting current upload run, and returning back in 5 secs
2013-02-26 15:20:41,548 Thread-832 DEBUG upload: Upload already running, aborting current upload run, and returning back in 5 secs
2013-02-26 15:20:46,555 Thread-833 DEBUG upload: Upload already running, aborting current upload run, and returning back in 5 secs

-  Although there are no communication issues between the Agent and the OMS machines, the output of 'emctl status agent' command shows that there is a pending upload backlog which keeps increasing:

./emctl status agent
......
Last successful upload : 2013-02-28 06:29:25
Total Megabytes of XML files uploaded so far : 372.27
Number of XML files pending upload : 241
Size of XML files pending upload(MB) : 42.72
Available disk space on upload filesystem : 59.77%
Last successful heartbeat to OMS : 2013-02-28 06:29:02
---------------------------------------------------------------
Agent is Running and Ready

When the size of upload backlog crosses the limit set for 98% of disk space, then the collections are also disabled.

-  At the Repository, executing the 'repvfy dump loader' shows:

1. 399. Severity processing too slow: 1

2. The statistics for '--- Severity Loader statistics per OMS (Last day)' shows that the severity loading has taken time ranging from 10 -18 mins in an hour.

3. The details for  'Number of incoming alerts per metric (last 4 hours)' shows:

TARGET_TYPE      METRIC_LABEL         PCT       CNT           CNT_CLEAR      CNT_WARN       CNT_CRIT
------------------  ------------               -------   ----------    ----------------- ----------------   ----------
oracle_database  Waits by Wait Class 95.89     133672      67914                    65758                  0
oracle_database  Database Limits        1.24        1734          765                       969                   0

This lists the metrics for which maximum number of alerts are being raised. The 'Waits by Wait Class' Database metric is responsible for 95% of the alerts raised in the past 4 hours. Also, the alerts being raised are mostly 'warning' and 'clear'.

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms