12.1.0.3 OMS Restarts with Job Dispatcher error: 31 JobWorkers are hung in pool Critical (Doc ID 1590566.1)

Last updated on JULY 09, 2021

Applies to:

Symptoms

12.1.0.3 OMS restarts very frequently with the following errors in <gc_inst>/em/EMGC_OMS1/sysman/log/emctl.msg file

HealthMonitor Oct 16, 2013 11:38:22 AM Job Dispatcher error: 31 JobWorkers are hung in pool Critical
Critical error err=3 detected in module Job Dispatcher
OMS will be restarted. A full thread dump will be generated in the log file
$OMS_INSTANCE_HOME/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs/EMGC_OMS1.out
to help Oracle Support analyse the problem.
Please consult My Oracle Support Note 964469.1 for additional information.

The <gc_inst>/em/EMGC_OMS1/sysman/log/emoms_pbs.trc shows the below errors repeated many times prior to the re-start:

2013-10-16 11:37:16,587 [Job Dispatcher Thread] ERROR emdrep.jobs logp.251 - [EM-00164] Too many job worker threads are hung for pool Critical.  31 workers are hung and 0 available out of 31 for the pool.
2013-10-16 11:37:18,099 [#EM_SYSTEM_POOL#:HealthMonitor] WARN  healthMonitor.HealthMonitor doWork.454 - HealthMonitor : Found errant task : TaskRegn:ID9718,Callback:class oracle.sysman.emdrep.jobs.JobWorker,Iterative:true,Duration:300,DueTime:1381937838089:Callback Details:Job Internal Pool:JobWorker Step -2 ->
Job: <internal>:<internal>
Step: bookkeeping_step (-2) [<internal>]
Command:wait_step[Internal, trusted]
JobId:0000000000000000 ExecId:0000000000000000 StepId:-2
Status: 1
StartTime: 2013-10-16 15:12:18.0
ReadyTime: 2013-10-16 15:12:18.0
Priority: 1

2013-10-16 11:37:24,680 [Job Dispatcher Thread] ERROR emdrep.jobs logp.251 - [EM-00155] Dispatcher cannot continue due to error "31 JobWorkers are hung in pool Critical". Forcing restart...

Cause

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.