The Pipeline Process Terminates Unexpectedly And Generates A 12GB Core File. (Doc ID 734702.1)

Last updated on SEPTEMBER 26, 2016

Applies to:

Oracle Communications Billing and Revenue Management - Version: 6.5.0.0.0 and later   [Release: 6.5.0 and later ]
Information in this document applies to any platform.
***Checked for relevance on 27-Dec-2011***

Symptoms

-- Problem Statement:
A core dump is generated when a ifw process terminates unexpectedly. The process log file shows a
CRITICAL error before it generates a core file.

26.06.2008 15:50:27 bri ifw IFW CRITICAL 00000 -
(ifw.Pipelines.RETAIL.Functions.Rating) Exception detected
in 'ifw.Pipelines.RETAIL.Functions.Rating'; thread id 35; Unrecognized exception.

After that, a "not enough space" message is showed for a few seconds before it crashed:

"Cannot open directory './xxxxxxx/pipelines/retail/data/in'; error message 'Not enough space'"

This machine has 8CPU and 20GB RAM Memory and the ifw process is using 12GB (average) everyday.
This issue has been occurring for the last five days and the pipeline crashed at least once per day.

Sometimes the pipeline process stopped without a core file but the problem still occurs with the
message:

“CRITICAL 00000 - (ifw.Pipelines.RETAIL.Functions.Rating) Exception detected in
'ifw.Pipelines.RETAIL.Functions.Rating'; thread id 35; Unrecognized exception.”

when the exception occurred it was written on the nohup file:

ifw: An error occurred during processing.

Originator : ifw.Pipelines.RETAIL.Functions.Rating
Destination :
Message : ERR_INVALID_THREAD_STATE
Arguments : Invalid thread state - abort processing

State : false
Severity : CRITICAL

After that message, the pipeline stopped.

-- Steps To Reproduce:
Start pipeline rating process to rate a huge number of CDR files. The ifw process crashed every 22
hours with the same message: Exception detected in 'ifw.Pipelines.RETAIL.Functions.Rating';
thread id 35; Unrecognized exception.

-- Business Impact:
The main impact is the discount file, after 5 times crashed, it can generate inconsistent data.

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms