Oracle Waveset Tasks Will Not Start. Many Tasks are Stuck in "Ready" or "Executing" State. Recon and ActiveSync Won't Start. (Doc ID 1017717.1)

Last updated on NOVEMBER 09, 2016

Applies to:

Oracle Waveset - Version 5.0 and later
All Platforms

Symptoms

The symptoms of this issue can present themselves in multiple forms.  You may see one, several, or all of the following symptoms:

  1. Reconcile and/or Activesync processes will not start and in the logs you see an error like this one: "Error starting <your adapter name here>.  Source adapter management is initializing".
  2. Many tasks in your system will not execute.  If you browse to the "Server Tasks->All Tasks" page, you will see many tasks in the "Ready" or "Executing" state, and a backlog may start to fill up that list.  You may have a few tasks that are getting processed, but most will not.
  3. Password Changes are not getting processed at all.
  4. You may not see "Reconciler" processes at all in your Server Tasks->All Tasks list, meaning the Reconciler process for your IDM servers won't even start up.

Changes

It is theorized that this issue is caused by a race condition between IDM Scheduler processes when two or more Schedulers attempt to lock a task at the same time, but a race condition is created due to some performance issue in the environment.  
Possible performance conditions which could cause this issue:
  1. Degraded database performance.  If db tables get too large or statistics are inaccurate, queries and update statements to the database could be slowed thereby causing a race condition between the two scheduler processes in our product as they attempt to lock tasks by writing a lock to the "task" table in the db.
  2. Too much load on the IDM servers.  This can be caused by a spike in activities like reconciliation, activesync, password changes or other custom workflows.  If CPU and/or memory utilization spikes dramatically and for too long, the schedulers will suffer degradation in performance and it could lead to this issue.
  3. IDM server clocks out of sync.  System clocks on each server must be synchronized.  This causes issues with the schedulers sharing time stamps between servers.   It is not likely that it would cause this task schedule corruption issue, but something to be aware of.  We recommend that you use some sort of NTP application to keep all server clocks sync'd.
  4. Network latency.  If there is a latency or outage on the network between one or both of the IDM servers and the repository database, corruption is likely.

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms