Demantra "Forecast Engine" Getting Stuck At Last Step with Notify notification ENG_MANAGER timeout 60 (Doc ID 2281085.1)

Last updated on JUNE 27, 2017

Applies to:

Oracle Demantra Demand Management - Version 12.2.5.1 and later
Information in this document applies to any platform.

Symptoms

On : 12.2.5.1 version, MICC Transfer to TSC

Customer mentions that they have applied patch 23756267 (12250219 ) and after that when they ran Forecast Engine.
It got stuck at the last step and it was running long.

VERSION BUILD SP UPGRADE_DATE
12.2.5 1 12250219 11-02-17 00:00:24
12.2.5 1 12250168 17-06-16 00:47:39
12.2.5 3 12250139 17-06-16 00:38:57
12.2.5 1 12250148 23-04-16 08:37:19
12.2.5 4 12250141 23-04-16 06:30:34
12.2.5 1 12250139 23-04-16 06:25:59
12.2.5 1 12250131 23-04-16 06:23:03
12.2.5 136 0 23-04-16 06:16:27
12.2.2 11 UPGRADED FROM 23-04-16 05:18:24
12.2.2 1 12220062 10-01-15 01:35:56
12.2.2 1 12220046 11-09-14 00:21:04
12.2.2 1 12220045 11-09-14 00:18:57
12.2.2 1 12220031 13-06-14 01:12:51

5th March they had run the Forecast Engine, It got stuck at last step more than an hour.
Later they cleared defunc and it got errors out at the last step.

Customer was seeing the error

09:54:24:011 EngineHeartBeatMonitor Stage: 2, Count: 2187, Prev Count: 2187, Time: 09:54:24, Prev time: 09:24:23
09:54:24:011 WARNING EngineHeartBeatMonitor detected frozen Engine process at stage: 2 for more than 1800 seconds
09:54:25:136 ERROR Engine about to abort due to severe error!!!

So they were asked to follow

Engine Error HRESULT=0X800706BE Remote Procedure Call Failed (Doc ID 1557175.1)


However that did not fix the issue

They keep getting issues where sometime the engine is a success and sometimes it fails

Demantra is deployed in 7 Linux boxes and the values are as below:

Box ENG_NUM_CONCURRENT
Linux box 1 : 4
Linux box 2 : 10
Linux box 3 : 10
Linux box 4 : 10
Linux box 5 : 10
Linux box 6 : 10
Linux box 7 : 10


A sample of Success and failures of the Forecast Engine:

3-Sep-16 Success
10-Sep-16 Success
17-Sep-16 Success
24-Sep-16 Failed SR 3-13384249951 : 24741710 Demantra "Forecast Engine" Getting stuck at last step
1-Oct-16 Failed
8-Oct-16 Success
15-Oct-16 Failed
22-Oct-16 Failed
29-Oct-16 Failed
5-Nov-16 Success
12-Nov-16 Failed
19-Nov-16 Failed
26-Nov-16 Success
3-Dec-16 Success
10-Dec-16 Success
17-Dec-16 Success
24-Dec-16 Failed
31-Dec-16 Success
7-Jan-17 Failed
14-Jan-17 Success
21-Jan-17 Failed
28-Jan-17 Failed
4-Feb-17 Failed
11-Feb-17 Success On 10-Feb => Patch: 23756267 applied
18-Feb-17 Success
25-Feb-17 Success
4-Mar-17 Failed
11-Mar-17 Success
18-Mar-17 Success
25-Mar-17 Success
1-Apr-17 Failed
8-Apr-17 Failed On 14-Apr-2017 as suggested TIMEOUT value in InitParams0Daily.xml was changed to 10000.
15-Apr-17 Failed

Engine logs show

08:41:09:802 WARNING [ManagerServiceListener] Got timeout while waiting for notifications. Continue.
08:41:39:806 WARNING [ManagerServiceListener] Got timeout while waiting for notifications. Continue.
08:42:00:915 Branch status - Progress(%): 98, ETA(min): 5, Elapsed(min): 260, Successfull: 252, Failed: 0, Remaining: 4
08:42:09:811 WARNING [ManagerServiceListener] Got timeout while waiting for notifications. Continue.
08:42:39:816 WARNING [ManagerServiceListener] Got timeout while waiting for notifications. Continue.
08:43:09:812 WARNING [ManagerServiceListener] Got timeout while waiting for notifications. Continue.
08:43:39:817 WARNING [ManagerServiceListener] Got timeout while waiting for notifications. Continue.
08:44:00:917 Branch status - Progress(%): 98, ETA(min): 5, Elapsed(min): 262, Successfull: 252, Failed: 0, Remaining: 4
08:44:01:107 Communication String

db_exception_log

Relevant Information Collection
---------------------------------------
ERR_DATE PROC_NAME ERR_MSG
18-APR-2017 20:12:57 NOTIFICATION.NOTIFY TIMEOUT Notify Msg : ConfirmationId=710396&EngineId=30&command=GetSimulationJob
18-APR-2017 20:12:57 NOTIFICATION.NOTIFY TIMEOUT Notify notification ENG_MANAGER timeout 60
18-APR-2017 20:12:57 NOTIFICATION.NOTIFY TIMEOUT Queue ENG_MANAGER_OUT_Q
18-APR-2017 20:12:53 NOTIFICATION.NOTIFY TIMEOUT Notify Msg : ConfirmationId=182778&EngineId=17&command=GetSimulationJob
18-APR-2017 20:12:53 NOTIFICATION.NOTIFY TIMEOUT Queue ENG_MANAGER_OUT_Q
18-APR-2017 20:12:53 NOTIFICATION.NOTIFY TIMEOUT Notify notification ENG_MANAGER timeout 60
18-APR-2017 20:12:53 NOTIFICATION.NOTIFY TIMEOUT Queue ENG_MANAGER_OUT_Q
18-APR-2017 20:12:53 NOTIFICATION.NOTIFY TIMEOUT Notify Msg : ConfirmationId=475003&EngineId=14&command=GetSimulationJob
18-APR-2017 20:12:53 NOTIFICATION.NOTIFY TIMEOUT Notify notification ENG_MANAGER timeout 60
18-APR-2017 20:12:53 NOTIFICATION.NOTIFY TIMEOUT Notify notification ENG_MANAGER timeout 60
18-APR-2017 20:12:53 NOTIFICATION.NOTIFY TIMEOUT Queue ENG_MANAGER_OUT_Q
18-APR-2017 20:12:53 NOTIFICATION.NOTIFY TIMEOUT Notify Msg : ConfirmationId=118287&EngineId=22&command=GetSimulationJob
18-APR-2017 20:12:52 NOTIFICATION.NOTIFY TIMEOUT Queue ENG_MANAGER_OUT_Q


Support has asked them to

1. Truncate the ENGINE_CACHES table

 Rerun PROPORT

 update mdp_matrix set prediction_status = 99, prop_changes=1;
 commit;
 exec proport;


2. The queues are valid

select *
from user_objects
where (object_name like 'DEM\_%\_IN%' ESCAPE '\' or object_name like
'ENG\_%\_IN%' ESCAPE '\')
or (object_name like 'DEM\_%\_OUT%' ESCAPE '\' or object_name like
'ENG\_%\_OUT%' ESCAPE '\')
and object_type in ('RULE SET','QUEUE','TABLE')

Returns all Valid

The below query returns no rows


select * from obj$
where obj# not in (
 select object_id
 from dba_objects);


3. Asked the customer backup the existing setenv.bat file on all the servers and change the ENG_NUM_CONCURRENT
variable to be 4.
  

But issue persists
ERROR
-----------------------
Long Running "Forecast Engine"




STEPS
-----------------------
The issue can be reproduced at will with the following steps:
1. Run the engine


BUSINESS IMPACT
-----------------------
The issue has the following business impact:
Due to this issue, users cannot get a forecast

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms