My Oracle Support Banner

Messaging Server 7.0.5 - Server Crash With Insufficient Memory - Generates Core Dump On Imexpire (Doc ID 2214600.1)

Last updated on JUNE 29, 2017

Applies to:

Oracle Communications Messaging Server - Version 7.0.5 and later
Information in this document applies to any platform.

Symptoms

Consider a setup running Messaging Server 7.0.5.35.0

Enabled with following sieve filter for imexpire to report on email having certain character in message  with 'From:' address match.


Rule1.folderpattern: user/%/INBOX
Rule1.sieve:allof(header :contains "Message-id" "@sampledomain.com", header :contains "From" "no-reply@gmail.com");
Rule1.action: report
Rule1.exclusive: 1


When applied this rule to all mailboxes, with a pattern of 'Inbox', encountered a number of core dumps, and Out-of-Memory situation. When looked at the created core dumps, pmap on the imexpire shows significantly high memory allocated.

When took a gcore from a fresh run with arguments, that quickly went up to 10GB:

imexpire -n -t1 -r1 -v 3 -f /tmp/header.imsrule

Observe that it picks up new memory quite fast and it seems to increase  memory usage while being busy handling user Inboxes (e.g. mapping in index data / digesting mails from that box):

# date; ps -ef -o vsz,pid |grep 19419

Friday, November 25, 2016 02:06:02 PM CET
13875528 19419

# date; ps -ef -o vsz,pid |grep 19419

Friday, November 25, 2016 02:06:03 PM CET
14002504 19419

# date; ps -ef -o vsz,pid |grep 19419

Friday, November 25, 2016 02:06:04 PM CET
14166344 19419

# ps -ef -o vsz,pid |grep 19419

Friday, November 25, 2016 02:06:08 PM CET
14606152 19419

# date; ps -ef -o vsz,pid |grep 19419

Friday, November 25, 2016 02:06:09 PM CET
14764296 19419

Did a follow up with '-v 3' omitted from the argument list as well as a reduced sieve filter (just a single test on message-id, no 'and' on 'To') as shown below:

Rule1.folderpattern: user/%/INBOX
Rule1.sieve:allof(header :contains "Message-id" "@sampledomain.com");
Rule1.action: report
Rule1.exclusive: 1


but processes still grow very quickly to above 6G.

According to '-v 3' output, it seems to take some several dozens to a few hundred users to reach significant (>4G) memory footprint.

When ran a quick test to see whether '-r1 -t1' exceeds 25G, saw just that happen. So even a 'sequential approach' seems to 'run away' in terms of memory usage.

When ran the imexpire against a single and different partition:

imexpire -n -p partition_one -t1 -r1  -f /tmp/header.imsrule

ps -ef -o vsz,pid,args |grep [e]xpire; date
10161768  5670 /msg-b/opt/sun/comms/messaging64/sbin/imexpire -n -p partition_one -t1 -r1 -f /t
Wednesday, November 30, 2016 09:12:44 AM CET

ps -ef -o vsz,pid,args |grep [e]xpire; date
10863720  5670 /msg-b/opt/sun/comms/messaging64/sbin/imexpire -n -p partition_one -t1 -r1 -f /t
Wednesday, November 30, 2016 09:12:49 AM CET

Another run with additional rss output:

ps -ef -o vsz,rss,pid,args |grep [e]xpire; date
622880 413560  7337 /opt/sun/comms/messaging64/sbin/imexpire -n -p partition_one -t1 -r1 -f /t
Wednesday, November 30, 2016 09:15:44 AM CET

ps -ef -o vsz,rss,pid,args |grep [e]xpire; date
1910928 1708856  7337 /opt/sun/comms/messaging64/sbin/imexpire -n -p partbunibi04 -t1 -r1 -f /t
Wednesday, November 30, 2016 09:15:53 AM CET

ps -ef -o vsz,rss,pid,args |grep [e]xpire; date
3289112 3092008  7337 /opt/sun/comms/messaging64/sbin/imexpire -n -p partbunibi04 -t1 -r1 -f /t
Wednesday, November 30, 2016 09:16:02 AM CET

ps -ef -o vsz,rss,pid,args |grep [e]xpire; date
4910584 4710440  7337 /opt/sun/comms/messaging64/sbin/imexpire -n -p partbunibi04 -t1 -r1 -f /t
Wednesday, November 30, 2016 09:16:13 AM CET

ps -ef -o vsz,rss,pid,args |grep [e]xpire; date
6368648 6170904  7337 /opt/sun/comms/messaging64/sbin/imexpire -n -p partbunibi04 -t1 -r1 -f /t
Wednesday, November 30, 2016 09:16:22 AM CET

As we can see rss and vsz raise pretty quickly and easily exceeds 10G.

Changes

 

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.