EM 12c 13c : emctl start agent Fails 'Starting agent ....Consult emctl.log and emagent.nohup in: /agent_inst/sysman/log' 'ERROR - Dead Lock detected!!' from gcagent.log (Doc ID 2170691.1)

Last updated on AUGUST 10, 2016

Applies to:

Enterprise Manager Base Platform - Version 12.1.0.1.0 to 12.1.0.3.0 [Release 12.1]
Information in this document applies to any platform.

Symptoms

$AGENT_INST/bin/emctl start agent

Oracle Enterprise Manager Cloud Control 12c Release 3  
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
Starting agent ...........................Consult emctl.log and emagent.nohup in: /u01/app/oraoem/agent12c/agent_inst/sysman/log

$AGENT_INST/sysman/log/gcagent_errors.log

2016-08-06 02:30:41,348 [42:GCThread-13] ERROR - Dead Lock detected!!
Participating threads:Thread Info Dump:
=================oraclldn.oracle.com=>[147044330820005])" tid=1442424 BLOCKED
  > Accumulated wait time (msec): 185020 (12 times)
  > Accumulated blocked time (msec): 3547608 (1 times)

"GC.Executor.6 (LOAD_TARGET for cluster.dldne403p-cl)" tid=113 BLOCKED
  > Accumulated wait time (msec): 2713401 (14 times)
  > Accumulated blocked time (msec): 1774070142 (61 times)

"CRSeOns" tid=1652 BLOCKED
  > Accumulated blocked time (msec): 1774070066 (3 times)
=================
Thread Info Dump:
=================
"HTTP Listener-1442424 - /emd/main/ (DispatchRequests OMS.pbs@14686@dldno601001por.oracle.com=>[147044330820005])" tid=1442424 BLOCKED
  oracle.sysman.db.receivelet.eons.EonsReceivelet.refreshState(EonsReceivelet.java:640)
  - blocked on <0x17fb52> (a java.lang.Object), which is owned by "GC.Executor.6 (LOAD_TARGET for cluster.dldne403p-cl)" (tid=113)
  oracle.sysman.gcagent.target.interaction.execution.ReceiveletFactory.refreshState(ReceiveletFactory.java:128)

$AGENT_INST/sysman/log/gcagent.log

"HTTP Listener-1432492 - /emd/main/ (DispatchRequests OMS.pbs@20662@dldno601002por.oracle.com=>[147038990860005])" tid=1432492 BLOCKED
 oracle.sysman.db.receivelet.eons.EonsReceivelet.refreshState(EonsReceivelet.java:640)
 - blocked on <0x17fb52> (a java.lang.Object), which is owned by "GC.Executor.6 (LOAD_TARGET for cluster.dldne403p-cl)" (tid=113)
 oracle.sysman.gcagent.target.interaction.execution.ReceiveletFactory.refreshState(ReceiveletFactory.java:128)
 oracle.sysman.gcagent.target.interaction.execution.ReceiveletManager.refreshState(ReceiveletManager.java:464)
 oracle.sysman.gcagent.target.interaction.execution.ReceiveletInteractionMgr.discardAllState(ReceiveletInteractionMgr.java:1371)
 oracle.sysman.gcagent.target.interaction.execution.ReceiveletInteractionMgr.discardState(ReceiveletInteractionMgr.java:1327)
 oracle.sysman.gcagent.metadata.impl.collection.CollectionManager.discardTargetState(CollectionManager.java:3503)
 oracle.sysman.gcagent.dispatch.cxl.DiscardStateAction.discardSeverityState(DiscardStateAction.java:132)
..............
 oracle.sysman.gcagent.comm.agent.http.HTTPListenerThreadPool$JobSleeve.run(HTTPListenerThreadPool.java:256)
 org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:436)
 oracle.sysman.gcagent.util.system.GCAThread$RunnableWrapper.run(GCAThread.java:184)
 java.lang.Thread.run(Thread.java:662)
 > Accumulated wait time (msec): 185020 (12 times)
 > Accumulated blocked time (msec): 57247930 (1 times)

"HTTP Listener-1431423 - /emd/main/ (DispatchRequests OMS.pbs@20662@dldno601002por.oracle.com=>[147038899630005])" tid=1431423 BLOCKED
 oracle.sysman.db.receivelet.eons.EonsReceivelet.refreshState(EonsReceivelet.java:640)
 - blocked on <0x17fb52> (a java.lang.Object), which is owned by "GC.Executor.6 (LOAD_TARGET for cluster.dldne403p-cl)" (tid=113)
 oracle.sysman.gcagent.target.interaction.execution.ReceiveletFactory.refreshState(ReceiveletFactory.java:128)
 oracle.sysman.gcagent.target.interaction.execution.ReceiveletManager.refreshState(ReceiveletManager.java:464)
 oracle.sysman.gcagent.target.interaction.execution.ReceiveletInteractionMgr.discardAllState(ReceiveletInteractionMgr.java:1371)
 oracle.sysman.gcagent.target.interaction.execution.ReceiveletInteractionMgr.discardState(ReceiveletInteractionMgr.java:1327)
 oracle.sysman.gcagent.metadata.impl.collection.CollectionManager.discardTargetState(CollectionManager.java:3503)
 oracle.sysman.gcagent.dispatch.cxl.DiscardStateAction.discardSeverityState(DiscardStateAction.java:132)
 oracle.sysman.gcagent.dispatch.cxl.DiscardStateAction.satisfyRequest(DiscardStateAction.java:89)
 oracle.sysman.gcagent.dispatch.ProcessRequestAction._call(ProcessRequestAction.java:134)
 oracle.sysman.gcagent.dispatch.ProcessRequestAction.call(ProcessRequestAction.java:95)
 oracle.sysman.gcagent.dispatch.InlineDispatchCoordinator.dispatchRequest(InlineDispatchCoordinator.java:229)
 oracle.sysman.gcagent.dispatch.DispatchRequestsAction.call(DispatchRequestsAction.java:110)
.......................
 org.eclipse.jetty.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:462)
 oracle.sysman.gcagent.comm.agent.http.HTTPListenerThreadPool$JobSleeve.run(HTTPListenerThreadPool.java:256)
 org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:436)
 oracle.sysman.gcagent.util.system.GCAThread$RunnableWrapper.run(GCAThread.java:184)
 java.lang.Thread.run(Thread.java:662)
 > Accumulated wait time (msec): 476181 (57 times)
 > Accumulated blocked time (msec): 58160298 (21 times)

..............
2016-08-06 02:37:50,759 [1923:A6770201:GC.SysExecutor.313 (AgentSystemMonitorTask)] WARN - Subsystem (Upload Manager) returned bad status of {+ Upload Manager: *Critical, mandatory component* +}
2016-08-06 02:37:50,761 [1923:A6770201] WARN - Subsystem (Ping Manager) returned bad status of {+ Ping Manager: *Critical, but not mandatory component* +}
2016-08-06 02:38:21,250 [1442661:HTTP Listener-1442661 - /emd/lifecycle/main/] ERROR - agent suffered internal error on request 147044750080001
2016-08-06 02:38:52,625 [1442661:HTTP Listener-1442661 - /emd/lifecycle/main/] ERROR - agent suffered internal error on request 147044753210001
2016-08-06 02:39:23,996 [1442661:HTTP Listener-1442661 - /emd/lifecycle/main/] ERROR - agent suffered internal error on request 147044756350001
2016-08-06 02:39:36,070 [1442663:7A8C33CA:Timer-266] INFO - Agent shutdown is misbehaved: self-destructing...
2016-08-06 02:39:36,070 [1442663:7A8C33CA] INFO - agent status is being changed to EXITING
2016-08-06 02:39:36,070 [1442663:7A8C33CA] INFO - Agent exiting with exit code 0
2016-08-06 02:39:36,071 [31:F9C26A76:Shutdown] INFO - *jetty*: Shutdown hook executing
2016-08-06 02:39:36,071 [31:F9C26A76] INFO - *jetty*: Graceful shutdown SslSelectChannelConnector@0.0.0.0:3872

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms