Cloud Control Management Server (OMS) may Crash When Visiting Oracle Scheduler -> Jobs Page for a Database target (Doc ID 2149943.1)

Last updated on MARCH 08, 2017

Applies to:

Enterprise Manager Base Platform - Version 12.1.0.4.0 and later
Enterprise Manager for Oracle Database - Version 12.1.0.7.0 and later
Information in this document applies to any platform.

Symptoms

Cloud Control Management Server may crash when the user is visiting Oracle Scheduler -> Jobs page for a database target. Stuck threads are observed on the WLS admin console for:

<Mar 21, 2016 8:43:52 AM CET> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '18' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "1,901" seconds working on the request "Workmanager: default, Version: 0, Scheduled=true, Started=true, Started time: 1901518 ms
[
GET /em/console/database/instance/schr/jobs?_em.coIFR=true&target=RACDB&type=rac_database HTTP/1.1
Accept: image/jpeg, application/x-ms-application, image/gif, application/xaml+xml, image/pjpeg, application/x-ms-xbap, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, */*
Referer: https://my.oms.com/em/faces/sdk/nonFacesWrapper?target=RACDB&_adf.ctrl-state=3g9c54ok5_199&_em.coBM=%2Fconsole%2Fdatabase%2Finstance%2Fschr%2Fjobs%3F_em.coIFR%3Dtrue%26target%3RACDB%26type%3Drac_database&type=rac_database&_afrLoop=34983942295371194&_afrWindowMode=0&_afrWindowId=3g9c54ok5_196
...

]", which is more than the configured time (StuckThreadMaxTime) of "1,800" seconds. Stack trace:

OMS logs will show:

HealthMonitor Mar 21, 2016 8:58:26 AM PbsAdminMsgListener error: PbsAdminMsgListener thread timed out.
Critical error err=3 detected in module PbsAdminMsgListener
OMS will be restarted. A full thread dump will be generated in the log file
/u01/app/oracle/product/12.1.0.5/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs/EMGC_OMS3.out
to help Oracle Support analyse the problem.

Also, the following indicators for this cause are available:

1. High CPU usage on the OMS server from the OMS processes

2. target database will show more than 2000 jobs being defined in the target database:

select count(*) from ALL_SCHEDULER_JOBS;

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms