My Oracle Support Banner

EM 12c: emctl upload agent 'EMD upload error:full upload has failed: Upload Recent Sending Statistics' Jstack shows 'org.apache.log4j.spi.RootLogger' Blocked Thread (Doc ID 2025759.1)

Last updated on APRIL 28, 2018

Applies to:

Enterprise Manager Base Platform - Version 12.1.0.4.0 and later
Information in this document applies to any platform.

Symptoms

EM 12c agent upload fails with following message

$/AGENT_INSt/bin/emctl upload agent

Oracle Enterprise Manager Cloud Control 12c Release 3
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
EMD upload error:full upload has failed: Upload Recent Sending Statistics
________________________________
Channel=data send[ms]=2925 fsz=439

Avg send time=2925.0 milliseconds
Backoff Event List
________________________________
Backoff event : channel=Metric Priority Data expiration at 2015-07-17 19:09:14 IST
Backoff event : channel=Metric Data expiration at 2015-07-17 19:08:53 IST
Upload Failure List
_______________________________
Connection Event List
_____________________________
Upload timed out before completion.
Number of files to upload before the uploadNow call: 463, total size (MB): 2.3745117
Remaining number of files to upload: 467, total size (MB): 2.4031343 (TIMEOUT)

From console Setup > Manage Cloud Console > Health Overview page shows 'Backoff Requests'

 

 

At Operating system , EMGC_OMS1 PID consuming ~190%+  CPU

 

Jstack of EMGC_OMS1 pid show  BLOCKED / RUNNABLE threads

[ACTIVE] ExecuteThread: '101' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00007ff43808c800 nid=0x6254 waiting for monitor entry [0x00007ff388975000]
 java.lang.Thread.State: BLOCKED (on object monitor)
      at org.apache.log4j.Category.callAppenders(Category.java:201)
      - waiting to lock <0x00000005fdc03dd0> (a org.apache.log4j.spi.RootLogger)
      at org.apache.log4j.Category.forcedLog(Category.java:388)
      at org.apache.log4j.Category.debug(Category.java:257)
      at oracle.sysman.core.pbs.receiver.OMSHandshake.handshake(OMSHandshake.java:1084)
      at oracle.sysman.core.pbs.receiver.AbstractOMSHandshake.handshake(AbstractOMSHandshake.java:174)
      at oracle.sysman.core.pbs.gcloader.Receiver.doGet(Receiver.java:703)
"GCLoader[metadata] - https://agt159.oracle.com:3872/emd/main/" daemon prio=10 tid=0x00007ff43807c000 nid=0x3a82 waiting for monitor entry [0x00007ff388c78000]
 java.lang.Thread.State: BLOCKED (on object monitor)
      at org.apache.log4j.Category.callAppenders(Category.java:201)
      - waiting to lock <0x00000005fdc03dd0> (a org.apache.log4j.spi.RootLogger)
      at org.apache.log4j.Category.forcedLog(Category.java:388)
      at org.apache.log4j.Category.log(Category.java:823)
      at oracle.sysman.util.logging.ApacheLogOperations.logp(ApacheLogOperations.java:251)
      at oracle.sysman.util.logging.DualModeLogOperations.logp(DualModeLogOperations.java:264)
      at oracle.sysman.emSDK.util.logging.Logger.logp(Logger.java:1032)
      at oracle.sysman.emSDK.util.logging.Logger.log(Logger.java:825)
      at oracle.sysman.emSDK.util.logging.Logger.fine(Logger.java:1353)
      
[ACTIVE] ExecuteThread: '101' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00007ff43808c800 nid=0x6254 waiting for monitor entry [0x00007ff388975000]
  java.lang.Thread.State: BLOCKED (on object monitor)
       at org.apache.log4j.Category.callAppenders(Category.java:201)
       - waiting to lock <0x00000005fdc03dd0> (a org.apache.log4j.spi.RootLogger)
       at org.apache.log4j.Category.forcedLog(Category.java:388)
       at org.apache.log4j.Category.debug(Category.java:257)
       at oracle.sysman.core.pbs.receiver.OMSHandshake.handshake(OMSHandshake.java:1084)
       at oracle.sysman.core.pbs.receiver.AbstractOMSHandshake.handshake(AbstractOMSHandshake.java:174)
       at oracle.sysman.core.pbs.gcloader.Receiver.doGet(Receiver.java:703)
"GCLoader[metadata] - https://agt259.oracle.com:3872/emd/main/" daemon prio=10 tid=0x00007ff43807c000 nid=0x3a82 waiting for monitor entry [0x00007ff388c78000]
  java.lang.Thread.State: BLOCKED (on object monitor)
       at org.apache.log4j.Category.callAppenders(Category.java:201)
       - waiting to lock <0x00000005fdc03dd0> (a org.apache.log4j.spi.RootLogger)
       at org.apache.log4j.Category.forcedLog(Category.java:388)
       at org.apache.log4j.Category.log(Category.java:823)
       at oracle.sysman.util.logging.ApacheLogOperations.logp(ApacheLogOperations.java:251)
       at oracle.sysman.util.logging.DualModeLogOperations.logp(DualModeLogOperations.java:264)
       at oracle.sysman.emSDK.util.logging.Logger.logp(Logger.java:1032)
       at oracle.sysman.emSDK.util.logging.Logger.log(Logger.java:825)
       at oracle.sysman.emSDK.util.logging.Logger.fine(Logger.java:1353)  

/gc_inst/em/EMGC_OMS#/sysman/log/emoms.trc show DEBUG level tracing

2015-07-17 23:30:54,652 [RemoteJobWorker 63047229] DEBUG cxl.PerformOperationResponse logp.251 - PerformOperationResponse Error="null"
2015-07-17 23:30:54,654 [RemoteJobWorker 63047229] DEBUG cxl.PerformOperationResponse logp.251 - PerformOperationResponse ExitCode=0
2015-07-17 23:30:54,657 [RemoteJobWorker 63047229] DEBUG cxl.PerformOperationResponse logp.251 - PerformOperationResponse Output="null"
2015-07-17 23:30:54,660 [RemoteJobWorker 63047229] DEBUG http.TMClientConnectionManager logp.251 - renewing connection to agent at https://igr.oracle.com:3872/emd/main/ for 3 MINUTES
2015-07-17 23:30:54,664 [RemoteJobWorker 63047229] DEBUG emdrep.jobs logp.251 - RemoteOp: asynchronous remote op dispatched
2015-07-17 23:30:54,678 [RemoteJobWorker 63047229] DEBUG emdrep.jobs entering.121 - Entering JobExecutionContextImpl.clearReposState()

 

 

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.