My Oracle Support Banner

EM 12c : emctl status agent, Agents are shown 'Unreachable in Console' 'STUCK Threads' Reported from emctl dump omsthread or jstack (Doc ID 1950895.1)

Last updated on APRIL 28, 2018

Applies to:

Enterprise Manager Base Platform - Version 12.1.0.1.0 and later
Information in this document applies to any platform.

Symptoms

Use Case 1

Many agents are uploading successfully to the OMS but the Agent is marked as Unreachable in the Console
The upload does not fail or report any errors

$/AGENT_INST/bin/emctl upload  agent

Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
EMD upload completed successfully

$/AGENT_INST/bin/emctl pingOMS                

Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
EMD pingOMS completed successfully

 

Loader errors from mgmt_system_error_log table in the repository database:

SQL>select occur_date, module_name, emd_url, error_msg  from mgmt_system_error_log where occur_date > sysdate - 10 order by occur_date desc;

09-NOV-14 10:42:00 LOADER https://euxhooradbpr.oracle.com:3872/e md/main/ UploadType: data File:2:65640.1415515320000 oracle.sysman.em l.ecm.track.RepositoryOperationException: executePreLoad: Al l metric collections failed, see metric collection errors: targetName : EMRDP-Infrastructure EMRDP2_euxhoorad bprodn2-Database Context targetType : oracle_apps_ inf collectionName : oracle_apps_inf_config targetGuid : 35CCE04F10A8BA5973C6A587351098C4 collection tim estamp : 2014-11-09 10:35:07
09-NOV-14 10:36:09 LOADER https://euxhooradbpr odn1.oracle.com:3872/e md/main/ UploadType: data File:2:75675.1415514967000 oracle.sysman.em l.ecm.track.RepositoryOperationException: executePreLoad: Al l metric collections failed, see metric collection errors: targetName : EMRDP-Infrastructure EMRDP1_euxhoorad bprodn1-Database Context targetType : oracle_apps_ inf collectionName : oracle_apps_inf_config targetGuid : E5947C7C3EC763E7FAF2878521D6B2A7 collection tim estamp : 2014-11-09 10:26:31
09-NOV-14 10:26:57 LOADER https://euxhooradbpr odn1.oracle.com:3872/e md/main/ UploadType: uploadonfetch File:1:75659.1415514417000 oracle. sysman.eml.ecm.track.RepositoryOperationException: executePr eLoad: All metric collections failed, see metric collection errors: targetName : EMRDP-Oracle E-Business Suit e Patch Information Configuration targetType : ora cle_apps_patch collectionName : oracle_apps_patch_conf ig targetGuid : A749B697C11306EECACFE3B5FA1977E0 c ollection timestamp : 2014-11-09 10:26:53
09-NOV-14 10:26:56 LOADER https://euxhooradbpr odn1.oracle.come:3872/e md/main/ UploadType: uploadonfetch File:1:75658.1415514416000 oracle. sysman.eml.ecm.track.RepositoryOperationException: executePr eLoad: All metric collections failed, see metric collection errors: targetName : EMRDP-Oracle E-Business Suit e Custom Objects Configuration targetType : oracle _apps_custom collectionName : oracle_apps_custom_confi g targetGuid : F6626972EC104A85A127E992804CE70E co llection timestamp : 2014-11-09 10:26:54
09-NOV-14 10:26:54 LOADER https://euxhooradbpr odn1.oracle.com:3872/e md/main/ UploadType: uploadonfetch File:0:75657.1415514414000 oracle. sysman.eml.ecm.track.RepositoryOperationException: executePr eLoad: All metric collections failed, see metric collection errors: targetName : EMRDP-Workflow Infrastructur e targetType : oracle_apps_wf collectionName : oracle_apps_wf_config targetGuid : 52A4FB7EE5F0 5E32C76EBF73FC3632CF collection timestamp : 2014-11-09 10:26 :53

Use Case 2

$/AGENT_INST/bin/emctl upload agent

fails with following error, where the Loader is sending a backoff request to this Agent

Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
EMD upload error:full upload has failed: Upload Recent Sending Statistics
________________________________

Avg send time=0.0 milliseconds
Backoff Event List
________________________________
Backoff event : channel=Response Severity expiration at 2014-11-13 15:11:35 BDT
Backoff event : channel=Response Severity expiration at 2014-11-13 15:11:31 BDT
Backoff event : channel=Response Severity expiration at 2014-11-13 15:11:40 BDT
Upload Failure List
_______________________________
Connection Event List
_____________________________
Upload timed out before completion.
Number of files to upload before the uploadNow call: 9000, total size (MB): 7.86                                            7359
Remaining number of files to upload: 9001, total size (MB): 7.867694 (TIMEOUT)

 OMS thread dump shows Java-level deadlock

$/OMS_HOME/bin/emctl dump omsthread

Found one Java-level deadlock:
=============================
"[STUCK] ExecuteThread: '30' for queue: 'weblogic.kernel.Default (self-tuning)'":
  waiting to lock monitor 0x000000000e630e50 (object 0x00000000dc6557f8, a java.lang.Class),
  which is held by "[STUCK] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'"
"[STUCK] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'":
  waiting to lock monitor 0x000000000c2bed08 (object 0x000000007502dbe0, a java.lang.Object),
  which is held by "Thread-950077"
"Thread-950077":
  waiting to lock monitor 0x000000000e630e50 (object 0x00000000dc6557f8, a java.lang.Class),
  which is held by "[STUCK] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'"

Java stack information for the threads listed above:
===================================================
"[STUCK] ExecuteThread: '30' for queue: 'weblogic.kernel.Default (self-tuning)'":
        at oracle.sysman.core.pbs.blackout.BlackoutCache.isTargetInBlackout(BlackoutCache.java:81)
        - waiting to lock <0x00000000dc6557f8> (a java.lang.Class for oracle.sysman.core.pbs.blackout.BlackoutCache)
        at oracle.sysman.core.pbs.blackout.BlackoutLoaderUtil.isTargetTimeInBlackoutWindow(BlackoutLoaderUtil.java:102)
        at oracle.sysman.core.pbs.gcloader.DataLoader.canProcessRecord(DataLoader.java:2177)
  
"Thread-950077":
        at oracle.sysman.core.pbs.blackout.BlackoutCache.setCacheData(BlackoutCache.java:156)
        - waiting to lock <0x00000000dc6557f8> (a java.lang.Class for oracle.sysman.core.pbs.blackout.BlackoutCache)
        at oracle.sysman.core.pbs.blackout.BlackoutCache.refreshBlackoutWindowCache(BlackoutCache.java:244)
        - locked <0x000000007502dbe0> (a java.lang.Object)
        at oracle.sysman.core.pbs.blackout.BlackoutCachePeriodicThread.run(BlackoutCachePeriodicThread.java:40)

Found 1 deadlock.

 

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


In this Document
Symptoms
 Use Case 1
 Use Case 2
Cause
Solution
References


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.