Previewing A Base View For Large Data Sets Causes The Dgraph To Consume Excessive Memory Affecting the Overall Performance Of The Endeca Server
Last updated on MAY 16, 2017
Applies to:Oracle Endeca Information Discovery Studio - Version 3.1.0 to 3.2.0 [Release 3.x]
Oracle Endeca Server - Version 7.6.0 to 7.7.0 [Release 7.x]
Information in this document applies to any platform.
Previewing Base Views in Oracle Endeca Information Discovery (OEID) Studio (Application Settings -> Views -> Base View -> Preview) causes Endeca Server to consume excessive memory in case of large data domains.
When Previewing the Base View for large data sets, the Dgraphs will grow to consume several times their disk size in memory.
This leads to paging of memory, which further affects the overall Performance of the Endeca Server.
Specifically, in case of large data sets (up to millions or records and up to 400 attributes) that take up 4GB, 12GB and 16GB on disk, they grow to 31GB, 55GB and 65GB in memory when Previewing the Base View.
The Preview will usually fail the first time around and when previewing a second time the dgraphs will grow to these sizes before returning the top 50 records.
As an example for the memory usage and disk access profiles, in the case of the 12GB data set, there was minimal disk access (between 100KB/s-300KB/s) while memory usage was growing significantly (500MB/s) and then the dgraphs would go through phases where disk usage would go down to 0 and memory usage would grow by 2GB/s for a while.
The servers are home to a number of dgraphs and during normal usage their memory usage does not reach these peaks, so there is RAM to spare at all times.
But when Previewing the Base View, this high-unbounded growth will more than exceed the remaining memory available, leading to paging memory to disk (further contributing to the performance issue).
When Previewing a Base View from Studio, the code returns a sample of 50 records and the total record count.
Previewing Base Views should not cause the dgraphs to consume such large amounts of memory.
The issue can be reproduced at will with the following steps:
1. Preview a Base View for a large data set (millions or records and hundreds of attributes).
The issue has the following business impact:
Due to this issue, users cannot use the other features of the product because the Endeca Server is overwhelmed by such high memory usage.
Sign In with your My Oracle Support account
Don't have a My Oracle Support account? Click to get started
My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms