EM 12c, 13c: Zombie Processes Prevent the Enterprise Manager Cloud Control Agent Monitoring FMW Targets
(Doc ID 2177106.1)
Last updated on NOVEMBER 13, 2019
Applies to:
Enterprise Manager Base Platform - Version 12.1.0.5.0 and laterInformation in this document applies to any platform.
Symptoms
The Enterprise Manager (EM) Cloud Control Agent is not starting due to zombie processed as noted in .../gc_inst/em/EMGC_OMS1/sysman/log/emagent.nohup:
INFO: incident 56 created with problem key oracle.sysman.gcagent.task.TaskZombieException:oracle.sysman.gcagent.task.TaskFutureImpl$WrappedTask:620, in directory \agent_inst\diag\ofm\emagent\emagent\incident\incdir_56
<Tue Aug 16 03:28:27 MDT 2016> <Warning> <RJVM> <WL-000510> <Unsolicited response: 5,084>
Aug 16, 2016 3:28:27 AM ClientNotifForwarder NotifFetcher-run
SEVERE: Failed to fetch notification, stopping thread. Error is: weblogic.rmi.extensions.RequestTimeoutException: RJVM response from 'weblogic.rjvm.RJVMImpl@6672fcea - id: myserver' connect time: 'Tue Aug 16 02:27:40 MDT 2016'' for 'fetchNotifications(JIJ)' timed out after: 30000ms.
at weblogic.rjvm.ResponseImpl.unmarshalReturn(ResponseImpl.java:237)
at weblogic.rmi.internal.BasicRemoteRef.invoke(BasicRemoteRef.java:223)
at javax.management.remote.rmi.RMIConnectionImpl_12130_WLStub.fetchNotifications(Unknown Source)
at weblogic.management.remote.common.RMIConnectionWrapper$26.run(ClientProviderBase.java:1229)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:143)
at weblogic.security.Security.runAs(Security.java:61)
at weblogic.management.remote.common.RMIConnectionWrapper.fetchNotifications(ClientProviderBase.java:1227)
at javax.management.remote.rmi.RMIConnector$RMINotifClient.fetchNotifs(RMIConnector.java:1306)
at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.fetchNotifs(ClientNotifForwarder.java:554)
at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.doRun(ClientNotifForwarder.java:437)
at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:418)
at com.sun.jmx.remote.internal.ClientNotifForwarder$LinearExecutor$1.run(ClientNotifForwarder.java:88)
<Tue Aug 16 03:28:27 MDT 2016> <Warning> <RJVM> <WL-000510> <Unsolicited response: 5,084>
Aug 16, 2016 3:28:27 AM ClientNotifForwarder NotifFetcher-run
SEVERE: Failed to fetch notification, stopping thread. Error is: weblogic.rmi.extensions.RequestTimeoutException: RJVM response from 'weblogic.rjvm.RJVMImpl@6672fcea - id: myserver' connect time: 'Tue Aug 16 02:27:40 MDT 2016'' for 'fetchNotifications(JIJ)' timed out after: 30000ms.
at weblogic.rjvm.ResponseImpl.unmarshalReturn(ResponseImpl.java:237)
at weblogic.rmi.internal.BasicRemoteRef.invoke(BasicRemoteRef.java:223)
at javax.management.remote.rmi.RMIConnectionImpl_12130_WLStub.fetchNotifications(Unknown Source)
at weblogic.management.remote.common.RMIConnectionWrapper$26.run(ClientProviderBase.java:1229)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:143)
at weblogic.security.Security.runAs(Security.java:61)
at weblogic.management.remote.common.RMIConnectionWrapper.fetchNotifications(ClientProviderBase.java:1227)
at javax.management.remote.rmi.RMIConnector$RMINotifClient.fetchNotifs(RMIConnector.java:1306)
at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.fetchNotifs(ClientNotifForwarder.java:554)
at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.doRun(ClientNotifForwarder.java:437)
at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:418)
at com.sun.jmx.remote.internal.ClientNotifForwarder$LinearExecutor$1.run(ClientNotifForwarder.java:88)
Reviewing the incident in the incident log files from EM Cloud Control:
Zombie processes are created for an EM Agent monitoring WebLogic targets. The incident dump has the following messaging:
(Stack Trace - c:\agent_inst\diag\ofm\emagent\emagent\incident\incdir_56\gcagent_error.log)
2016-08-16 02:27:34,320 [13709:GC.Executor.11825 (weblogic_j2eeserver:/<application>/myserver:application_metric)]
ERROR - weblogic_j2eeserver:/<application>/myserver:application_metric
oracle.sysman.gcagent.task.TaskZombieException: task declared as a zombie
at oracle.sysman.gcagent.task.TaskFutureImpl$WrappedTask.accountedCall(TaskFutureImpl.java:620)
at oracle.sysman.gcagent.task.TaskFutureImpl$WrappedTask.call(TaskFutureImpl.java:643)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at oracle.sysman.gcagent.task.TaskFutureImpl.run1(TaskFutureImpl.java:380)
at oracle.sysman.gcagent.task.TaskFutureImpl.run(TaskFutureImpl.java:337)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at oracle.sysman.gcagent.task.executor.TrackThreadFactory$1.run(TrackThreadFactory.java:54)
at oracle.sysman.gcagent.util.system.GCAThread$RunnableWrapper.run(GCAThread.java:189)
at java.lang.Thread.run(Thread.java:662)
ERROR - weblogic_j2eeserver:/<application>/myserver:application_metric
oracle.sysman.gcagent.task.TaskZombieException: task declared as a zombie
at oracle.sysman.gcagent.task.TaskFutureImpl$WrappedTask.accountedCall(TaskFutureImpl.java:620)
at oracle.sysman.gcagent.task.TaskFutureImpl$WrappedTask.call(TaskFutureImpl.java:643)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at oracle.sysman.gcagent.task.TaskFutureImpl.run1(TaskFutureImpl.java:380)
at oracle.sysman.gcagent.task.TaskFutureImpl.run(TaskFutureImpl.java:337)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at oracle.sysman.gcagent.task.executor.TrackThreadFactory$1.run(TrackThreadFactory.java:54)
at oracle.sysman.gcagent.util.system.GCAThread$RunnableWrapper.run(GCAThread.java:189)
at java.lang.Thread.run(Thread.java:662)
(Stack Trace - c:\agent_inst\diag\ofm\emagent\emagent\incident\incdir_56\EM_AGENT_InMemoryLog4_j56.txt)
2016-08-16 02:27:34,362 [13709:GC.Executor.11825] DEBUG - End task weblogic_j2eeserver:/<application>/myserver:application_metric
2016-08-16 02:27:34,363 [28:IncCreHandler] DEBUG - Creating an incident: task declared as a zombie (1380592633)
2016-08-16 02:27:34,363 [28:IncCreHandler] DEBUG - Creating an incident: task declared as a zombie (1380592633)
Cause
To view full details, sign in with your My Oracle Support account. |
|
Don't have a My Oracle Support account? Click to get started! |
In this Document
| Symptoms |
| Cause |
| Solution |