My Oracle Support Banner

The Voucher and Wallet Server (VWS) Pair is Slowly and Silently Getting Out Of Sync and Some Transactions are being Silently Reverted (Doc ID 1264891.1)

Last updated on MAY 25, 2018

Applies to:

Oracle Communications Network Charging and Control - Version: 2.4.0 and later   [Release: 2.4 and later ]
Information in this document applies to any platform.
This issue affects all the following Patches:
BE Patch 43287 (1.6) v2.4.0.17
BE Patch 44508 (1.1) v2.4.0.18
BE Patch 46850 (1.1) v2.4.0.19
BE Patch 46048 (1.5) v2.4.0.20
BE Patch 94102 (1.1) v2.4.0.21
BE Patch 98911 (1.3) v2.4.0.22

Symptoms

Symptom 1

Over time, the VWS pair's bucket information is slowly getting out of sync. This can be seen by manually running the following script on the SMS server as the ccs_oper user:

ccs_oper@SMS:/IN/service_packages/CCS$ cd /IN/service_packages/CCS/VW_SyncTools/Wallets

ccs_oper@SMS:/IN/service_packages/CCS/VW_SyncTools/Wallets$ ./check_be_wallet_bucket_diffs.sh
Select a UBE pair from the following list:
1) VWS01 [351] / VWS02 [352] ID: 01
UBE Pair[1-1]? 1

UBE Pair: 1
Domain ID: 01
Primary: VWS01 [db@"E2BE_01.WORLD"]
Secondary: VWS02 [db@"E2BE_02.WORLD"]


    WALLET         ID VALUE (PRI) VALUE (SEC) DIFF (PRI-SEC)
---------- ---------- ----------- ----------- --------------
      1519       6610       92899       92890              9

The above output shows that for Wallet 1519, there is a Bucket with ID 6610 which is 9 units greater on the Primary than the Secondary VWS server. If the buckets are getting out of sync, the number of rows returned will increase over time.

By scheduling the execution of the aforementioned script and gathering enough data for definitive trend analysis, a graph such as the following would be an indication that there is likely a problem:


It is important to note that increased subscriber activity may show trends like the graph above. In this case, it is worth comparing the wallet unsync rate with the Call Attempts per Second (CAPS) and Short Message Attempts per Minute (SMAPS). If the CAPS and SMAPS remain constant over a period of a few days, but the VWS' continue to get further out of sync, then it is likely this problem may be affecting the VWS'.



Symptom 2

From the SMS EDR Screen or the raw VWS Event Data Records (EDR), events indicate a successful billing event has occurred, but the next sqeuential EDR for the same subscriber shows the old balance information. Take for example, the following VWS EDRs:

BILLING_ENGINE_ID=1|SCP_ID=87783972|SEQUENCE_NUMBER=0|CDR_TYPE=5|RECORD_DATE=20101106083140|ACCT_ID=1519|ACCT_REF_ID=1519
|WALLET_TYPE=7|ACS_CUST_ID=12|CS=S|TCS=20101106083140|BALANCE_TYPES=22|BALANCES=6993|COSTS=9|ACCOUNT_TYPE=7|EVENT_CLASS=VPLMN
|EVENT_NAME=IntDirectories|EVENT_COST=9|EVENT_TIME_COST=0.00|EVENT_DATA_COST=0|EVENT_UNIT_COST=1|EVENT_COUNT=1|DISCOUNT=0
|CASCADE=81|OLD_BALANCE_EXPIRIES=0|NEW_BALANCE_EXPIRIES=0

BILLING_ENGINE_ID=1|SCP_ID=87783972|SEQUENCE_NUMBER=0|CDR_TYPE=5|RECORD_DATE=20101106083113|ACCT_ID=1519|ACCT_REF_ID=1519
|WALLET_TYPE=7|ACS_CUST_ID=12|CS=S|TCS=20101106083113|BALANCE_TYPES=22|BALANCES=6993|COSTS=9|ACCOUNT_TYPE=7|EVENT_CLASS=VPLMN
|EVENT_NAME=IntDirectories|EVENT_COST=9|EVENT_TIME_COST=0.00|EVENT_DATA_COST=0|EVENT_UNIT_COST=1|EVENT_COUNT=1|DISCOUNT=0
|CASCADE=81|OLD_BALANCE_EXPIRIES=0|NEW_BALANCE_EXPIRIES=0

BILLING_ENGINE_ID=1|SCP_ID=87783972|SEQUENCE_NUMBER=0|CDR_TYPE=5|RECORD_DATE=20101106083111|ACCT_ID=1519|ACCT_REF_ID=1519
|WALLET_TYPE=7|ACS_CUST_ID=12|CS=S|TCS=20101106083111|BALANCE_TYPES=22|BALANCES=6993|COSTS=9|ACCOUNT_TYPE=7|EVENT_CLASS=VPLMN
|EVENT_NAME=IntDirectories|EVENT_COST=9|EVENT_TIME_COST=0.00|EVENT_DATA_COST=0|EVENT_UNIT_COST=1|EVENT_COUNT=1|DISCOUNT=0
|CASCADE=81|OLD_BALANCE_EXPIRIES=0|NEW_BALANCE_EXPIRIES=0

BILLING_ENGINE_ID=1|SCP_ID=87783972|SEQUENCE_NUMBER=0|CDR_TYPE=5|RECORD_DATE=20101106082813|ACCT_ID=1519|ACCT_REF_ID=1519
|WALLET_TYPE=7|ACS_CUST_ID=12|CS=S|TCS=20101106082813|BALANCE_TYPES=22|BALANCES=6993|COSTS=9|ACCOUNT_TYPE=7|EVENT_CLASS=VPLMN
|EVENT_NAME=IntDirectories|EVENT_COST=9|EVENT_TIME_COST=0.00|EVENT_DATA_COST=0|EVENT_UNIT_COST=1|EVENT_COUNT=1|DISCOUNT=0
|CASCADE=81|OLD_BALANCE_EXPIRIES=0|NEW_BALANCE_EXPIRIES=0

Things to note:
  1. All the RECORD_DATE tag values are different (1 second apart in this case) which means this is not duplicate EDRs
  2. The ACCT_ID and ACCT_REF_ID are the same so it is the same subscriber in all the EDRs
  3. BALANCE_TYPES is always 22, so we are accessing the same Balance Type for the subscriber
  4. The BALANCES is always 6993 and the COSTS is always 9
This information implies that each charge attempt (for the cost of 9 units) is not being committed.

An alternate way to view this evidence is via the SMS Screens for a suspected impacted subscriber. When viewing the EDRs for the same Balance Type, the Old Balance value will not change between two rows, despite a positive Cost value. For example:

Changes

Increase in traffic load or adjustments to the parameters which control the flushing of Wallet information from the beVWARS cache.

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.