12.1.0.3 OMS Restarts with Job Dispatcher error: 31 JobWorkers are hung in pool Critical (Doc ID 1590566.1)

Last updated on FEBRUARY 02, 2016

Applies to:

Enterprise Manager Base Platform - Version 12.1.0.3.0 to 12.1.0.3.0 [Release 12.1]
Information in this document applies to any platform.

Symptoms


 

12.1.0.3 OMS restarts very frequently with the following errors in <gc_inst>/em/EMGC_OMS1/sysman/log/emctl.msg file

HealthMonitor Oct 16, 2013 11:38:22 AM Job Dispatcher error: 31 JobWorkers are hung in pool Critical
Critical error err=3 detected in module Job Dispatcher
OMS will be restarted. A full thread dump will be generated in the log file
/a01/app/oracle/product/12.1.0/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs/EMGC_OMS1.out
to help Oracle Support analyse the problem.
Please consult My Oracle Support Note 964469.1 for additional information.

The <gc_inst>/em/EMGC_OMS1/sysman/log/emoms_pbs.trc shows the below errors repeated many times prior to the re-start:

2013-10-16 11:37:16,587 [Job Dispatcher Thread] ERROR emdrep.jobs logp.251 - [EM-00164] Too many job worker threads are hung for pool Critical.  31 workers are hung and 0 available out of 31 for the pool.
2013-10-16 11:37:18,099 [#EM_SYSTEM_POOL#:HealthMonitor] WARN  healthMonitor.HealthMonitor doWork.454 - HealthMonitor : Found errant task : TaskRegn:ID9718,Callback:class oracle.sysman.emdrep.jobs.JobWorker,Iterative:true,Duration:300,DueTime:1381937838089:Callback Details:Job Internal Pool:JobWorker Step -2 ->
Job: <internal>:<internal>
Step: bookkeeping_step (-2) [<internal>]
Command:wait_step[Internal, trusted]
JobId:0000000000000000 ExecId:0000000000000000 StepId:-2
Status: 1
StartTime: 2013-10-16 15:12:18.0
ReadyTime: 2013-10-16 15:12:18.0
Priority: 1

2013-10-16 11:37:24,680 [Job Dispatcher Thread] ERROR emdrep.jobs logp.251 - [EM-00155] Dispatcher cannot continue due to error "31 JobWorkers are hung in pool Critical". Forcing restart...

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms