Messaging Server 7.0.5 - Server Crash With Insufficient Memory - Generates Core Dump On Imexpire (Doc ID 2214600.1)

Last updated on DECEMBER 05, 2019

Applies to:

Symptoms

Consider a setup running Messaging Server 7.0.5.35.0

Enabled with following sieve filter for imexpire to report on email having certain character in message  with 'From:' address match.

When applied this rule to all mailboxes, with a pattern of 'Inbox', encountered a number of core dumps, and Out-of-Memory situation. When looked at the created core dumps, pmap on the imexpire shows significantly high memory allocated.

When took a gcore from a fresh run with arguments, that quickly went up to 10GB:

imexpire -n -t1 -r1 -v 3 -f /tmp/header.imsrule

Observe that it picks up new memory quite fast and it seems to increase  memory usage while being busy handling user Inboxes (e.g. mapping in index data / digesting mails from that box):

# date; ps -ef -o vsz,pid |grep 19419

Friday, November 25, 2016 02:06:02 PM CET
13875528 19419

# date; ps -ef -o vsz,pid |grep 19419

Friday, November 25, 2016 02:06:03 PM CET
14002504 19419

# date; ps -ef -o vsz,pid |grep 19419

Friday, November 25, 2016 02:06:04 PM CET
14166344 19419

# ps -ef -o vsz,pid |grep 19419

Friday, November 25, 2016 02:06:08 PM CET
14606152 19419

# date; ps -ef -o vsz,pid |grep 19419

Friday, November 25, 2016 02:06:09 PM CET
14764296 19419

Did a follow up with '-v 3' omitted from the argument list as well as a reduced sieve filter (just a single test on message-id, no 'and' on 'To') as shown below:

but processes still grow very quickly to above 6G.

According to '-v 3' output, it seems to take some several dozens to a few hundred users to reach significant (>4G) memory footprint.

When ran a quick test to see whether '-r1 -t1' exceeds 25G, saw just that happen. So even a 'sequential approach' seems to 'run away' in terms of memory usage.

When ran the imexpire against a single and different partition:

imexpire -n -p partition_one -t1 -r1  -f /tmp/header.imsrule

ps -ef -o vsz,pid,args |grep [e]xpire; date
10161768  5670 /opt/sun/comms/messaging64/sbin/imexpire -n -p partition_one -t1 -r1 -f /t
Wednesday, November 30, 2016 09:12:44 AM CET

ps -ef -o vsz,pid,args |grep [e]xpire; date
10863720  5670 /opt/sun/comms/messaging64/sbin/imexpire -n -p partition_one -t1 -r1 -f /t
Wednesday, November 30, 2016 09:12:49 AM CET

Another run with additional rss output:

ps -ef -o vsz,rss,pid,args |grep [e]xpire; date
622880 413560  7337 /opt/sun/comms/messaging64/sbin/imexpire -n -p partition_one -t1 -r1 -f /t
Wednesday, November 30, 2016 09:15:44 AM CET

ps -ef -o vsz,rss,pid,args |grep [e]xpire; date
1910928 1708856  7337 /opt/sun/comms/messaging64/sbin/imexpire -n -p partition_one -t1 -r1 -f /t
Wednesday, November 30, 2016 09:15:53 AM CET

ps -ef -o vsz,rss,pid,args |grep [e]xpire; date
3289112 3092008  7337 /opt/sun/comms/messaging64/sbin/imexpire -n -p partition_one -t1 -r1 -f /t
Wednesday, November 30, 2016 09:16:02 AM CET

ps -ef -o vsz,rss,pid,args |grep [e]xpire; date
4910584 4710440  7337 /opt/sun/comms/messaging64/sbin/imexpire -n -p partition_one -t1 -r1 -f /t
Wednesday, November 30, 2016 09:16:13 AM CET

ps -ef -o vsz,rss,pid,args |grep [e]xpire; date
6368648 6170904  7337 /opt/sun/comms/messaging64/sbin/imexpire -n -p partition_one -t1 -r1 -f /t
Wednesday, November 30, 2016 09:16:22 AM CET

As we can see rss and vsz raise pretty quickly and easily exceeds 10G.

Cause

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.