My Oracle Support Banner

NoSQL Persistent Files Not Getting Deleted (Doc ID 2747843.1)

Last updated on FEBRUARY 01, 2021

Applies to:

Oracle Communications BRM - Elastic Charging Engine - Version 11.3.0.6.0 and later
Information in this document applies to any platform.

Goal

On NoSQL version 12.1.3.5.2, and Elastic Charging Engine (ECE) version 11.3.0.6.22, Production system:

A situation was encountered wherein NoSQL persistent files are not getting deleted, and rpool (nosql file system) reached 100%.

Getting the below exception from the NoSQL logs:


Below are more details about the sequence of events that happened during the incident:

1. On a certain day at 08:47 PM, NoSQL persistent files (*.jdb) files stopped getting deleted from all three NoSQL servers
2. Each JDB file of around 1GB size piled up in the server and on next day at 05:00 PM, ZFS mountpoint (filesystem) capacity reached 100%
3. Custom alerts in filesystem capacity monitoring for 70% and 85% was not sent (probably operation team have not configured this correctly)
4. To avoid impacting rating engine (ECE), usage rating was stopped on 17/Dec/2020 at 05:00 PM
5. UNIX admin team increased the filesystem (rpool) by 100GB for all three NoSQL servers
6. Once filesystem capacity came down to 79%, nosql persistent files got deleted
7. After 2hrs, system came back to normalcy

Question:

What caused NoSQL persistent files to stop getting deleted in above scenario?
 
 

Solution

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


In this Document
Goal
Solution


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.