Batch Jobs Not Running Correctly And Remain 'Pending' After Applying Patch (Doc ID 1533662.1)

Last updated on DECEMBER 11, 2023

Applies to:

Symptoms

After deploying two patches and restarting the environment & threadpoolworkers (TPWs) the customer is having trouble with the batch jobs. The same patches were deployed in a test environment with no problem.  When restarting the TPWs, sometimes all the jobs may not start back up properly.

As a workaround, the customer would shut down the TPWs, update status to error/ended in all the batch related tables, truncate tspace table and then restart. Sometimes, it's necessary to truncate all the MWM batch related tables. However, today nothing would work. The jobs would restart, but then remain in ‘Pending’ status in CI_BATCH_JOB table while some others in ‘Pending’ status in CI_BATCH_THD table.  This affected random jobs at random times. One job M1-TSKTR just refuses to run. It always remains in 'Pending' status in CI_BATCH_JOB table. The workaround has been tried several times with no success. The jobs are running fine in a dev environment. This is happening only in the Prod environment.

For the logs, it looks like there are not enough threads available to schedule jobs for execution. In the beginning, there were 9 threads available:
.
-  2013-02-19 22:33:03,258 [pool-1-thread-3] INFO  (support.cluster.ClusteredNode) processNewWork available threads: 9 service: DEFAULT

and this is reduced to 2 and eventually none available:

-  2013-02-19 22:33:04,299 [pool-1-thread-2] INFO  (support.cluster.ClusteredNode) processNewWork available threads: 2 service: DEFAULT
-  2013-02-19 22:33:04,299 [pool-1-thread-2] DEBUG (support.cluster.ClusteredNode) processNewWork  scheduling thread:  BatchThread_Id(batchRunId: [batchControlId: [M1-SHWKT], batchNumber: 1,
batchRerunNumber: 0], batchThreadNumber: 1)
-  2013-02-19 22:33:04,301 [DEFAULTWorker:0] INFO  (support.batch.BatchWorkInSessionExecutable) Closing log writer
-  2013-02-19 22:33:04,306 [pool-1-thread-2] DEBUG (support.cluster.BatchClusterCache) chose member: 17
-  2013-02-19 22:33:04,307 [pool-1-thread-2] DEBUG (support.cluster.ClusteredNode) caught exception: commonj.work.WorkException: Servers are not available message: Servers are not available

Cause

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.