EM 12c: Zombie Processes Prevent the Enterprise Manager 12c Cloud Control Agent Monitoring FMW Targets from Starting (Doc ID 2177106.1)

Last updated on SEPTEMBER 03, 2016

Applies to:

Enterprise Manager Base Platform - Version 12.1.0.5.0 and later
Information in this document applies to any platform.

Symptoms

The Enterprise Manager (EM) 12.1.0.5 Cloud Control Agent is not starting due to zombie processed as noted in .../gc_inst/em/EMGC_OMS1/sysman/log/emagent.nohup:

 

INFO: incident 56 created with problem key oracle.sysman.gcagent.task.TaskZombieException:oracle.sysman.gcagent.task.TaskFutureImpl$WrappedTask:620, in directory c:\oemagent12c_slc09ofe\agent_inst\diag\ofm\emagent\emagent\incident\incdir_56
<Tue Aug 16 03:28:27 MDT 2016> <Warning> <RJVM> <WL-000510> <Unsolicited response: 5,084>
Aug 16, 2016 3:28:27 AM ClientNotifForwarder NotifFetcher-run
SEVERE: Failed to fetch notification, stopping thread. Error is: weblogic.rmi.extensions.RequestTimeoutException: RJVM response from 'weblogic.rjvm.RJVMImpl@6672fcea - id: '-2388346915601509119S:10.196.39.54:[-1,-1,6500,-1,-1,-1,6500]:splapp:myserver' connect time: 'Tue Aug 16 02:27:40 MDT 2016'' for 'fetchNotifications(JIJ)' timed out after: 30000ms.
weblogic.rmi.extensions.RequestTimeoutException: RJVM response from 'weblogic.rjvm.RJVMImpl@6672fcea - id: '-2388346915601509119S:10.196.39.54:[-1,-1,6500,-1,-1,-1,6500]:splapp:myserver' connect time: 'Tue Aug 16 02:27:40 MDT 2016'' for 'fetchNotifications(JIJ)' timed out after: 30000ms.
at weblogic.rjvm.ResponseImpl.unmarshalReturn(ResponseImpl.java:237)
at weblogic.rmi.internal.BasicRemoteRef.invoke(BasicRemoteRef.java:223)
at javax.management.remote.rmi.RMIConnectionImpl_12130_WLStub.fetchNotifications(Unknown Source)
at weblogic.management.remote.common.RMIConnectionWrapper$26.run(ClientProviderBase.java:1229)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:143)
at weblogic.security.Security.runAs(Security.java:61)
at weblogic.management.remote.common.RMIConnectionWrapper.fetchNotifications(ClientProviderBase.java:1227)
at javax.management.remote.rmi.RMIConnector$RMINotifClient.fetchNotifs(RMIConnector.java:1306)
at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.fetchNotifs(ClientNotifForwarder.java:554)
at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.doRun(ClientNotifForwarder.java:437)
at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:418)
at com.sun.jmx.remote.internal.ClientNotifForwarder$LinearExecutor$1.run(ClientNotifForwarder.java:88)

 

Reviewing the incident in the incident log files from EM 12.1.0.5 Cloud Control:

Zombie processes are created for an EM Agent monitoring WebLogic targets. The incident dump has the following messaging:

(Stack Trace - c:\oemagent12c_slc09ofe\agent_inst\diag\ofm\emagent\emagent\incident\incdir_56\gcagent_error.log)

2016-08-16 02:27:34,320 [13709:GC.Executor.11825 (weblogic_j2eeserver:/<application>/splapp/myserver:application_metric)]
ERROR - weblogic_j2eeserver:/<application>/splapp/myserver:application_metric
oracle.sysman.gcagent.task.TaskZombieException: task declared as a zombie
at oracle.sysman.gcagent.task.TaskFutureImpl$WrappedTask.accountedCall(TaskFutureImpl.java:620)
at oracle.sysman.gcagent.task.TaskFutureImpl$WrappedTask.call(TaskFutureImpl.java:643)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at oracle.sysman.gcagent.task.TaskFutureImpl.run1(TaskFutureImpl.java:380)
at oracle.sysman.gcagent.task.TaskFutureImpl.run(TaskFutureImpl.java:337)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at oracle.sysman.gcagent.task.executor.TrackThreadFactory$1.run(TrackThreadFactory.java:54)
at oracle.sysman.gcagent.util.system.GCAThread$RunnableWrapper.run(GCAThread.java:189)
at java.lang.Thread.run(Thread.java:662)

 (Stack Trace - c:\\agent_inst\diag\ofm\emagent\emagent\incident\incdir_56\EM_AGENT_InMemoryLog4_j56.txt)

2016-08-16 02:27:34,362 [13709:GC.Executor.11825] DEBUG - End task weblogic_j2eeserver:/<application>/splapp/myserver:application_metric
2016-08-16 02:27:34,363 [28:IncCreHandler] DEBUG - Creating an incident: task declared as a zombie (1380592633)

 

 

Changes

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms