Pipeline Rating Performance Issue In BRM 7.5 PS5 (Doc ID 1966740.1)

Last updated on SEPTEMBER 02, 2023

Applies to:

Symptoms

On BRM 7.5PS5, Pipeline Performance

Issue:

The user was rating at a regular normal CDR/sec rate until migrated from a legacy system of a special set of corporate accounts with a completely different structure (hierarchies, number of services per account, number of service login aliases, etc).

The performance dropped from 83 CDRs/sec to 23 CDRs/sec.

Until facing this problem, the user was operating the system with acceptable operational times in every process (rating, billing, AR, and invoicing). So far, the accounts were mostly residential, with no account hierarchies, and a moderate number of telephony services per account.

This week, another subset of accounts that represent part of the corporate clients has been migrated. This time, the account structure is completely different.

These corporate accounts have the following hierarchy structure:

One Parent Account - No services, products, or discounts; 1 Balance Group; 1 Billinfo.
One to many Child accounts - No services, products, or discounts; 1 Balance Group; 1 Billinfo.
One to many Grand Children Accounts - One to many (thousands) services per account; Many subscription groups per account; 1 Balance group per account (containing from 1 to thousands services linked to them); 1 Billinfo per account.

As you can see, this is a very heterogeneous group of account "types". It's normal that the performance is affected but not at this rate, specially having in mind that they have dedicated a much "bigger" server to rate these clients CDRs than the residential ones, and still they obtain a 4 to 1 performance decrease.

The volume of accounts, services, and even CDRs is very low, adding to the concern.

These are some metrics of the system including all the business segments, not only these new corporate ones:
- 184k active accounts
- 177k active telephony services (the only ones receiving CDRs)
- Aprox. 300k-400k CDRs rated per residential accounts per day with discounts. These are being rated in a separate (smaller) server in aprox. 1 hour.
- Aprox. 80k-90k CDRs rated per the problematic corporate accounts WITHOUT discounts.
- There are 10172 active telephony services in these problematic accounts.
- The maximum number of active services per grand children account is 4388, the second 3440, the third 1171 and the rest is below 500 to 10
- The maximum number of login alias per telephony service is 2999, the second 999, 699, 149 and so on. Only 25 services have login aliases.

The CDR files for these corporate accounts have been isolated, meaning that the most powerful server is in use to rate corporate CDRs exclusively.

In other words, the server is mostly idle while the Pipeline is working, there are plenty of memory locks inside the Pipeline process due to threads.

Changes

Cause

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.