12.1.0.3 OMS Restarts with Job Dispatcher error: 31 JobWorkers are hung in pool Critical
(Doc ID 1590566.1)
Last updated on JULY 09, 2021
Applies to:
Enterprise Manager Base Platform - Version 12.1.0.3.0 to 12.1.0.3.0 [Release 12.1]Information in this document applies to any platform.
Symptoms
12.1.0.3 OMS restarts very frequently with the following errors in <gc_inst>/em/EMGC_OMS1/sysman/log/emctl.msg file
HealthMonitor Oct 16, 2013 11:38:22 AM Job Dispatcher error: 31 JobWorkers are hung in pool Critical
Critical error err=3 detected in module Job Dispatcher
OMS will be restarted. A full thread dump will be generated in the log file
$OMS_INSTANCE_HOME/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs/EMGC_OMS1.out
to help Oracle Support analyse the problem.
Please consult My Oracle Support Note 964469.1 for additional information.
The <gc_inst>/em/EMGC_OMS1/sysman/log/emoms_pbs.trc shows the below errors repeated many times prior to the re-start:
2013-10-16 11:37:16,587 [Job Dispatcher Thread] ERROR emdrep.jobs logp.251 - [EM-00164] Too many job worker threads are hung for pool Critical. 31 workers are hung and 0 available out of 31 for the pool.
2013-10-16 11:37:18,099 [#EM_SYSTEM_POOL#:HealthMonitor] WARN healthMonitor.HealthMonitor doWork.454 - HealthMonitor : Found errant task : TaskRegn:ID9718,Callback:class oracle.sysman.emdrep.jobs.JobWorker,Iterative:true,Duration:300,DueTime:1381937838089:Callback Details:Job Internal Pool:JobWorker Step -2 ->
Job: <internal>:<internal>
Step: bookkeeping_step (-2) [<internal>]
Command:wait_step[Internal, trusted]
JobId:0000000000000000 ExecId:0000000000000000 StepId:-2
Status: 1
StartTime: 2013-10-16 15:12:18.0
ReadyTime: 2013-10-16 15:12:18.0
Priority: 1
2013-10-16 11:37:24,680 [Job Dispatcher Thread] ERROR emdrep.jobs logp.251 - [EM-00155] Dispatcher cannot continue due to error "31 JobWorkers are hung in pool Critical". Forcing restart...
Cause
To view full details, sign in with your My Oracle Support account. |
|
Don't have a My Oracle Support account? Click to get started! |
In this Document
Symptoms |
Cause |
Solution |
References |