Pipeline ONLINE_PRE_PROCESS Rate of File Processing Getting Reduced (Doc ID 2730644.1)

Last updated on NOVEMBER 24, 2020

Applies to:

Goal

How to improve the start-up and pre-processor Pipeline processing to handle large number of files?

The functional modules performance with instrumentation looks to be normal; however, ONLINE_PRE_PROCESS Pipeline is taking 40 minutes to process last 15K files (of around 300k files batch) with total of 120K records.

For e.g.
Run1 - 15000 files with each 53 records - 6 mins
Run2 - 15000 files with each 53 records - 8 mins
Run3 - 15000 files with each 53 records - 10 mins
Run4 - 15000 files with each 53 records - 12 mins

Additionally, one of the functional module MergeDSG_Fulfillment is taking more memory and more time to load during startup. This module is present in all of the Online Pipelines, which takes 1 hour plus for 116K records in iRule referred data file. Module load took 10 minutes if it present in single Pipeline and time is getting incremented based on the number of references.
 

Solution

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.