Under Certain Circumstances Messages Are Never Retrieved From Om_automation_message (Doc ID 2814770.1)

Last updated on MARCH 29, 2023

Applies to:

Symptoms

In OSM 7.3.5, there is a new feature which enables handling incoming messages received at a time when the order is in a state preventing normal task processing (amending) by storing the messages received in such circumstances in the DB table om_automation_message from where they are retrieved again when the order transitions from state amending back into state in_progress which again enables updating order data and transitioning tasks other than the ones related to compensation generated by revision.

The customer noticed the mechanism under certain circumstances appeared to fail in its purpose and consequently messages were left in the om_automation_message table after the order transitioned back from amending to in_progress state, and the related tasks which would  otherwise be finalized by the affected messages remained in accepted state and hence the orders were stuck.


By studying the problem in detail we determined the exact chain of events leading to the phenomenon described above. Revision is received, order remains in in_progess state since there is at least one task in accepted state, which prevents the state transition. Message is received on a JMS queue associated to an external event receiver automator. Before running the automator, the automation framework seems to check whether the order is in a compatible state, but since compensation has not started yet and the order at that moment is in in_progress state, the automator execution will start.

Grace period is reached on the task(s) which were blocking the transition of the order affected order from in_progress to amending state, consequently the order state changes.  The execution of automator discussed in item 2 above reaches the point where order is updated and / or order data updated. There is exception thrown and caught by the automation framework, before the message is marked for reprocessing after re-delivery interval, it is somehow marked with a flag which means that when reprocessed, it will be stored in om_automation_message table.  Compensation stemming from amendment discussed in item 1 above completes and the order moves back from amending to in_progess state. The message discussed in item 4 above is not stored in om_automation_message table yet, because re-delivery interval is not reached yet.

Re-delivery interval is reached and message is stored in om_automation_message table instead of being processed by the respective automator in the usual fashion. Unless there is another revision, the message is never re-queued and removed from the DB table and the order remains in the task which was to be completed by the message in question. The chain of events described above typically happens very fast and it is not possible to reproduce this under normal circumstances. Therefore, in order to be able to reproduce the problem, we introduced artificially delays in the automator execution by including the followng instruction in the xquery:

You will see trace in log file similar to the one below:
[2021-07-13T13:03:05.440+00:00] [AdminServer] [TRACE] []
[oracle.communications.ordermanagement.automation.message.MessagePersistenceManager] [tid:
[ACTIVE].ExecuteThread: '40' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: comdev] [ecid:
b2ff8239-e890-4028-a88f-7b029ccc5ac2-00001cc0,0] [APP: oms] [partition-name: DOMAIN] [tenant-
name: GLOBAL] [DSID: 0000NeVijDNFw0K6yVqYMG1WvM0M00000d] [SRC_CLASS:
oracle.communications.ordermanagement.automation.message.MessagePersistenceManager]
[SRC_METHOD: checkConditions] Checked condition
[CancelledOrchestrationOrderDiscardCondition:false]
[2021-07-13T13:03:05.441+00:00] [AdminServer] [TRACE] []
[oracle.communications.ordermanagement.automation.message.MessagePersistenceManager] [tid:
[ACTIVE].ExecuteThread: '40' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: comdev] [ecid:
b2ff8239-e890-4028-a88f-7b029ccc5ac2-00001cc0,0] [APP: oms] [partition-name: DOMAIN] [tenant-
name: GLOBAL] [DSID: 0000NeVijDNFw0K6yVqYMG1WvM0M00000d] [SRC_CLASS:
oracle.communications.ordermanagement.automation.message.MessagePersistenceManager]
[SRC_METHOD: checkConditions] Checked condition [OrderStateTransitionUnparkCondition:true]
Once the redelivery interval set on the JMS queue „de/telefonica/bug/response“ is
reached after the exception discussed in step 10, the following trace is logged in the
diagnostic log:
[2021-07-13T13:04:47.739+00:00] [AdminServer] [TRACE] []
[oracle.communications.ordermanagement.automation.message.MessagePersistenceManager] [tid:
[ACTIVE].ExecuteThread: '16' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: oms-internal]
[ecid: b2ff8239-e890-4028-a88f-7b029ccc5ac2-0000000a,0:410:1:43:713] [APP: oms] [partition-name:
DOMAIN] [tenant-name: GLOBAL] [SRC_CLASS:
oracle.communications.ordermanagement.automation.message.MessagePersistenceManager]
[SRC_METHOD: isEnabled] Checking message parking enabled [enabled:true]
[2021-07-13T13:04:47.739+00:00] [AdminServer] [TRACE] []
[oracle.communications.ordermanagement.automation.message.MessagePersistenceManager] [tid:
[ACTIVE].ExecuteThread: '16' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: oms-internal]
[ecid: b2ff8239-e890-4028-a88f-7b029ccc5ac2-0000000a,0:410:1:43:713] [APP: oms] [partition-name:
DOMAIN] [tenant-name: GLOBAL] [SRC_CLASS:
oracle.communications.ordermanagement.automation.message.MessagePersistenceManager]
[SRC_METHOD: isParkingCandidate] Checking whether message is marked for parking [marked:true
[2021-07-13T13:04:47.741+00:00] [AdminServer] [TRACE] []
[oracle.communications.ordermanagement.automation.message.MessagePersistenceManager] [tid:
[ACTIVE].ExecuteThread: '16' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: oms-internal]
[ecid: b2ff8239-e890-4028-a88f-7b029ccc5ac2-0000000a,0:410:1:43:713] [APP: oms] [partition-name:
DOMAIN] [tenant-name: GLOBAL] [SRC_CLASS:
oracle.communications.ordermanagement.automation.message.MessagePersistenceManager]
[SRC_METHOD: storeMessage] Storing message in database
[orderSeqId:107;histSeqId:203;correlationId:107;automationId:90008;cartridgeId:24;originalMessageId
:ID:]
Now check the database for entry in table om_automation_message:
Now we are in a situation where the message will never be retrieved unless there is another
amendment; the problem is reproduced.

Changes

Cause

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.