NoSQL Persistent Files Not Getting Deleted
(Doc ID 2747843.1)
Last updated on FEBRUARY 01, 2021
Applies to:Oracle Communications BRM - Elastic Charging Engine - Version 188.8.131.52.0 and later
Information in this document applies to any platform.
On NoSQL version 184.108.40.206.2, and Elastic Charging Engine (ECE) version 220.127.116.11.22, Production system:
A situation was encountered wherein NoSQL persistent files are not getting deleted, and rpool (nosql file system) reached 100%.
Getting the below exception from the NoSQL logs:
Below are more details about the sequence of events that happened during the incident:
1. On a certain day at 08:47 PM, NoSQL persistent files (*.jdb) files stopped getting deleted from all three NoSQL servers
2. Each JDB file of around 1GB size piled up in the server and on next day at 05:00 PM, ZFS mountpoint (filesystem) capacity reached 100%
3. Custom alerts in filesystem capacity monitoring for 70% and 85% was not sent (probably operation team have not configured this correctly)
4. To avoid impacting rating engine (ECE), usage rating was stopped on 17/Dec/2020 at 05:00 PM
5. UNIX admin team increased the filesystem (rpool) by 100GB for all three NoSQL servers
6. Once filesystem capacity came down to 79%, nosql persistent files got deleted
7. After 2hrs, system came back to normalcy
What caused NoSQL persistent files to stop getting deleted in above scenario?
To view full details, sign in with your My Oracle Support account.
Don't have a My Oracle Support account? Click to get started!
In this Document