OMS 10.2.0.5 - Successive Metadata Upload Requests Have Failed."Agent should retry later.ORA-20609 Error" obtaining upload lock on emd_url (Doc ID 1190183.1)

Last updated on JANUARY 02, 2015

Applies to:

Enterprise Manager Base Platform - Version 10.2.0.5 to 10.2.0.5 [Release 10.2]
Information in this document applies to any platform.

Symptoms

Multi 10.2.0.5.2 OMS with 11.1.0.7.0 Repository Database.

Host is shown as Agent unreachable and noticed the following error message in
OMS_HOME/sysman/emoms.trc file

"9 successive metadata upload requests have failed. Last metadata upload error is OMS already processing a Metadata or Severity file from this Agent. Agent should retry later.Error obtaining upload lock in OMS on emd_url:http://agent_name.domain:3872/emd/main/
Metric=Consecutive metadata upload-agent_name.domain failure count "


10gR5 RAC Agents upload is failing with following message:

2010-06-21 17:57:14,010 Thread-3751730064 ERROR upload: Failed to upload file B0004370.xml: HTTP error.
Response received: ERROR-400|OMS already processing a Metadata or Severity file from this Agent. Agent should retry later.ORA-20609: Error obtaining upload lock on emd_url:http://agent_name.domain:3872/emd/main/
ORA-06512: at "SYSMAN.EMD_LOADER", line 4564
ORA-06512: at line 1



From OMS_HOME/sysman/log/emoms.trc

2010-06-18 22:32:12,714 [AJPRequestHandler-ApplicationServerThread-31] ERROR eml.FxferRecv doGet.969 - Agent with url http://agent_name.domain:3872/emd/main/  is out-of-sync with repository (2/20) !
2010-06-18 23:21:53,234 [AJPRequestHandler-ApplicationServerThread-12] ERROR eml.FxferRecv doGet.969 - Agent with url http://agent_name.domain:3872/emd/main/ is out-of-sync with repository (2/5) !


Found deadlock at repository Database , snippet from alert_<SID>.log

Mon Jun 21 18:56:10 2010
Global Enqueue Services Deadlock detected. More info in file
/oracle/logfiles/diag/rdbms/oemdb/oemdb2/trace/oemdb2_lmd0_2543.trc.
Mon Jun 21 18:59:37 2010
Incremental checkpoint up to RBA [0x8f9b.15102.0], current log tail at RBA
[0x8f9b.1d211.0]
Mon Jun 21 19:02:30 2010
AUD: Audit Commit Delay exceeded, written a copy to OS Audit Trail


From oemdb2_lmd0_2543.trc

valblk : 0x3a423135202c204e554d5f4348494c44 :B15 , NUM_CHILD
DUMP LOCAL BLOCKER: initiate state dump for DEADLOCK
possible owner[119.3018] on resource TX-002E0004-000288DB

*** 2010-06-21 18:56:09.611
Submitting asynchronized dump request [28]
Global blockers dump end:-----------------------------------
Global Wait-For-Graph(WFG) at ddTS[0.355] :
BLOCKED 0x3501a11d0 5 wq 2 cvtops x1 TX 0x270005.0x54af0
[3D000-0002-00000326] 1
BLOCKER 0x3501a1020 5 wq 1 cvtops x28 TX 0x270005.0x54af0
[77000-0002-00000054] 1
BLOCKED 0x351575af8 5 wq 2 cvtops x1 TX 0x2e0004.0x288db
[77000-0002-00000054] 1
BLOCKER 0x350b85748 5 wq 1 cvtops x28 TX 0x2e0004.0x288db
[3D000-0002-00000326] 1
* Cancel deadlock victim lockp 0x3501a11d0
kjddt2vb: valblk [0.357] > local ts [0.356]
*********************


From oemdb2_lmd0_2543.trc

*** 2010-06-18 23:26:44.403
user session for deadlock lock 0x353a51d90
sid: 3250 ser: 6 audsid: 6956551 user: 53/SYSMAN flags: 0x8000045
pid: 55 O/S info: user: oracle, term: UNKNOWN, ospid: 4573
image: oracleorkxpoedb02.esp.aur.national.com.au
client details:
O/S info: user: , term: , ospid: 1234
machine: OMS_name.domainname program: OMS
client info: OMS_Name.somain:4889_Management_Service
application name: OEM.SystemPool, hash value=2960518376
action name: JobDispatcher, hash value=875884737
current SQL:
UPDATE MGMT_JOB_HISTORY SET JOB_ID=:B29 , EXECUTION_ID=:B28 , STEP_ID=:B27 ,
SOURCE_STEP_ID=:B26 , ORIGINAL_STEP_ID=:B25 , RESTART_MODE=:B24 ,
STEP_NAME=:B23 , STEP_TYPE=:B22 , COMMAND_TYPE=:B21 , ITERATE_PARAM=:B20 ,
ITERATE_PARAM_INDEX=:B19 , PARENT_STEP_ID=:B18 , STEP_STATUS=:B17 ,
STEP_STATUS_CODE=:B16 , STEP_STATUS_CODE_CATEGORY=:B15 , NUM_CHILDREN=:B14 ,
NUM_CHILDREN_COMPLETED=:B13 , OUTPUT_ID=:B12 , ERROR_ID=:B11 ,
START_TIME=:B10 , END_TIME=:B9 , SEQUENCE_NUMBER=:B8 , DISPATCHER_ID=:B7 ,
OMS_NAME=:B6 , STATUS_DETAIL= :B5 , STOP_START_TIME=:B4 ,
STOP_LAST_SCHEDULED_TIME= :B3 , STOP_ERROR_ID=:B2 , STOPPED_BY=:B1 WHERE
STEP_ID=:B27

.
*** 2010-06-19 05:45:03.046
user session for deadlock lock 0x3515707b8
sid: 3194 ser: 4582 audsid: 6958621 user: 53/SYSMAN flags: 0x8000045
pid: 29 O/S info: user: oracle, term: UNKNOWN, ospid: 10408
image: oracleorkxpoedb02.esp.aur.national.com.au
client details:
O/S info: user: , term: , ospid: 1234
machine: OMS_name.domain program: OMS
client info: OMS_name.domain:4889_Management_Service
application name: OEM.CacheModeWaitPool, hash value=796036576
current SQL:

DELETE FROM MGMT_JOB_EXECUTION WHERE STEP_ID = :B1



Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms