MSONWL64 Consumes Memory Very Quickly And Hangs on Slice Stage for ECC Constrained WITH Detailed Scheduling ASCP Plan (Doc ID 1532473.1)

Last updated on NOVEMBER 01, 2022

Applies to:

Symptoms

On : 12.1.3 version, SCO in DEV:

Users on 12.1.3 and have applied 12.1.3.8 VCP patch 14247039. They also have applied one off patch 15881048 which gave them MSONWL64.exe 120.3.12010000.274. They're running an ECC Constrained Plan With Detailed Scheduling. However it hangs in Slice #62 and then starts chewing up memory until memory is almost exhausted but then it just hangs without failing. They have 14 gig free and 8 gig swap space. We saw it use almost all of both base memory and swap and not write anything more in the log file. This is a non debug log file:

MSONWL64 module: Memory Based Planner 64-bit Linux

/dcmpic/mtlog/DCMPIC_vmohscmpi003/logs/appl/conc/out/data2025/
The above is the location of the data files

Plan stopped here at the bottom of the log file:

Slice # 61 (57 demands) from priority: 4 to: 4
TIMER: slice 60 pre phase 1 time = 0
All Operation = 258
TIMER: slice 60 phase 1 time = 0
TIMER: in phase 2, non firmed ops computeexceptions for demand 5237 time = 0
TIMER: in phase 2, non firmed ops computeexceptions for demand 7309 time = 0
TIMER: in phase 2, non firmed ops computeexceptions for demand 7642 time = 0
TIMER: in phase 2, non firmed ops computeexceptions for demand 6430 time = 0
TIMER: in phase 2, non firmed ops computeexceptions for demand 6828 time = 0
TIMER: in phase 2, non firmed ops computeexceptions for demand 6225 time = 0
TIMER: in phase 2, non firmed ops computeexceptions for demand 4079 time = 0
TIMER: in phase 2, non firmed ops computeexceptions for demand 4753 time = 0
TIMER: slice 60 phase 2 time = 0
TIMER: slice 60 phase 2 time = 0
TIMER: slice 60 phase 3 time = 0
Memory Used: 14.0715 MB; Scheduling duration: 0.00183333 Mins
====================================================
Slice # 62 (18 demands) from priority: 4 to: 4
TIMER: slice 61 pre phase 1 time = 41
All Operation = 226690
TIMER: slice 61 phase 1 time = 808

From a debug log file where we let it run for an hour it just starts printing what appears to be millions of rows with the following and just keep writing the following:

TRACE OM vendor id: 8436 site id: -23453
TRACE OM vendor id: 8436 site id: -23453
TRACE OM vendor id: 8436 site id: -23453
TRACE OM vendor id: 8436 site id: -23453
TRACE OM vendor id: 8436 site id: -23453
TRACE OM vendor id: 8436 site id: -23453
TRACE OM vendor id: 8436 site id: -23453
TRACE OM vendor id: 8436 site id: -23453

Before the plan was launched top command

$ top
top - 11:54:47 up 17:22, 9 users, load average: 1.00, 1.02, 1.13
Tasks: 306 total, 2 running, 302 sleeping, 0 stopped, 2 zombie
Cpu(s): 6.0%us, 0.9%sy, 0.0%ni, 91.9%id, 1.1%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 16777216k total, 1893684k used, 14883532k free, 20960k buffers
Swap: 8388600k total, 882380k used, 7506220k free, 381040k cached

 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3456 apdcmpic 25 0 93872 45m 15m R 99.8 0.3 29:31.40 frmweb
19856 apdcmpic 16 0 551m 25m 2264 S 2.0 0.2 0:06.40 java
  1 root 15 0 10348 456 420 S 0.0 0.0 0:00.06 init
  2 root RT -5 0 0 0 S 0.0 0.0 0:00.62 migration/0
  3 root 34 19 0 0 0 S 0.0 0.0 0:00.16 ksoftirqd/0

This shows 16 gig available 8 gig of swap. Avallable base is 14gig and 7 swap

This is waht it looked like when MSONWL64 was running:

top - 10:59:26 up 16:27, 8 users, load average: 1.24, 1.35, 1.17
Tasks: 301 total, 2 running, 297 sleeping, 0 stopped, 2 zombie
Cpu(s): 24.8%us, 2.0%sy, 0.0%ni, 72.8%id, 0.3%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 16777216k total, 16742764k used, 34452k free, 30304k buffers
Swap: 8388600k total, 8385672k used, 2928k free, 371892k cached

 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4395 apdcmpic 25 0 20.9g 13g 1596 R 98.4 85.8 159:12.03 MSONWL64.exe
22456 aptcmpic 15 0 67388 884 688 S 5.3 0.0 4:06.87 httpd
19307 apdcmpic 15 0 68412 892 688 S 4.7 0.0 4:10.78 httpd
30140 kavery 15 0 12872 1240 800 R 0.7 0.0 0:00.06 top
3577 odmagent 16 0 362m 35m 6156 S 0.3 0.2 2:28.01 emagent
17975 apdcmpic 16 0 12888 1240 804 S 0.3 0.0 0:15.23 top
  1 root 15 0 10348 456 420 S 0.0 0.0 0:00.06 init

Took all base and swap memory and just got stuck

Background:

Customer running plans 06-FEB-2013 and all was well. Then they found they had some sourcing rules setup incorrectly between organizations. Then 07-FEB-2007 with the new sourcing rules in place, the plan hangs. We did check all the logs for loops and found no sourcing rule loops. The Snapshot and workers complete in less than 5 minutes but the MSONWL64 just hangs when running ECC Constrained With Detailed Scheduling

ASCP plan is constrained with detailed scheduling (SCO type plan)

The issue began on 06-FEB-2013. We had been running a Constrained WITH Detailed Scheduling plan before that without any problems but we noticed there were some sourcing rules that were setup as a BUY FROM a supplier when they should be TRANSFER FROM an organization (Demands from the distribution organizations where not being passed onto the production plants). We made this change on 05-FEB-2013 and after collecting Sourcing Rules and running the plan that is when the issue began. The plan we used remained unchanged as it was already set up as would be required and it already had the Assignment Set selected.

This could lead us to consider the Sourcing Rules as a possible cause for the problem if some where causing circular supply references. However, yesterday I launched an Unconstrained plan with all other plan options the same as before (except the ones related to restrictions and decision rules) which completed OK in under 20 minutes so this shows that it mustn't be caused by a circular reference, most likely. After that, I launched a Classic Constrained plan which is still running after more than 12 hours.

Relating to changes to supply chain data that is all I can think of at the moment.

Important - Unconstrained plan finished as expected but because they're using process effectivity it doesn't pick up the right recipe (OPM) and hence they don't get the planning they want.

Also They tried Classic ECC Constrained plan didn't complete either after running 24 hours. So hangup in SCO perhaps:


EXPECTED BEHAVIOR
-----------------------
While the plan would not be expected to complete in 10 minutes when unconstrained, we would not expect it to run for more than 1 hour or less and not chew up all the available memory

STEPS
-----------------------
The issue can be reproduced at will with the following steps:
1. Login to Application.
2. Run ECC Constrained With Detailed Scheduling
  NAV: Advanced Supply Chain Planner (R) -> Supply Chain Plan -> Launch
3. See that the MBP takes hangs
4. Unconstrained plan takes about 10 minutes to run and complete.

Cause

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.