My Oracle Support Banner

Oracle Linux: "Out of memory: Kill process xxxxx (splunkd)" Entry In /var/log/messages Log on a System Running with Splunk (Doc ID 2478339.1)

Last updated on APRIL 04, 2019

Applies to:

Linux OS - Version Oracle Linux 6.9 to Oracle Linux 7.6 [Release OL6U9 to OL7U6]
Linux x86-64
Linux x86

Symptoms

  1. Oracle Linux running with Splunk.

  2. Further details are found in /var/log/messages:

    Nov 27 12:53:23 localhost kernel: : Out of memory: Kill process 14134 (splunkd) score 87 or sacrifice child
    Nov 27 12:53:23 localhost kernel: : Killed process 14134, UID 8500, (splunkd) total-vm:17361344kB, anon-rss:356kB, file-rss:264kB
    Nov 27 12:53:23 localhost kernel: : NET: Registered protocol family 36
    Nov 27 12:53:23 localhost kernel: : adclient invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0, oom_score_adj=0
    Nov 27 12:53:23 localhost kernel: : adclient cpuset=/ mems_allowed=0
    Nov 27 12:53:23 localhost kernel: : Pid: 2764, comm: adclient Not tainted 2.6.32-754.6.3.el6.x86_64 #1
    Nov 27 12:53:23 localhost kernel: : Call Trace:
    Nov 27 12:53:23 localhost kernel: : [<ffffffff81136670>] ? dump_header+0x90/0x1b0
    Nov 27 12:53:23 localhost kernel: : [<ffffffff81244d82>] ? security_real_capable_noaudit+0x42/0x70
    Nov 27 12:53:23 localhost kernel: : [<ffffffff81136af2>] ? oom_kill_process+0x82/0x2a0
    Nov 27 12:53:23 localhost kernel: : [<ffffffff81136a31>] ? select_bad_process+0xe1/0x120
    Nov 27 12:53:23 localhost kernel: : [<ffffffff81136f30>] ? out_of_memory+0x220/0x3c0
    Nov 27 12:53:23 localhost kernel: : [<ffffffff811439f1>] ? __alloc_pages_nodemask+0x941/0x960
    Nov 27 12:53:23 localhost kernel: : [<ffffffff8117ed8a>] ? alloc_pages_vma+0x9a/0x150
    Nov 27 12:53:23 localhost kernel: : [<ffffffff811718e2>] ? read_swap_cache_async+0xf2/0x160
    Nov 27 12:53:23 localhost kernel: : [<ffffffff81172479>] ? valid_swaphandles+0x69/0x160
    Nov 27 12:53:23 localhost kernel: : [<ffffffff811719d7>] ? swapin_readahead+0x87/0xc0
    Nov 27 12:53:23 localhost kernel: : [<ffffffff8115fa72>] ? handle_pte_fault+0x302/0xc80
    Nov 27 12:53:23 localhost kernel: : [<ffffffff810708c0>] ? wake_up_state+0x10/0x20
    Nov 27 12:53:23 localhost kernel: : [<ffffffff810bed5c>] ? wake_futex+0x3c/0x60
    Nov 27 12:53:23 localhost syslog-ng[30890]: syslog-ng starting up; version='3.2.5'
    Nov 27 12:53:23 localhost kernel: : [<ffffffff811606f6>] ? handle_mm_fault+0x306/0x450
    Nov 27 12:53:23 localhost kernel: : [<ffffffff81559821>] ? _cond_resched+0x1/0x40
    Nov 27 12:53:23 localhost kernel: : [<ffffffff81054db1>] ? __do_page_fault+0x141/0x500
    Nov 27 12:53:23 localhost kernel: : [<ffffffff81069380>] ? __dequeue_entity+0x30/0x50
    Nov 27 12:53:23 localhost kernel: : [<ffffffff81558d78>] ? schedule+0x4a8/0xcb0
    Nov 27 12:53:23 localhost kernel: : [<ffffffff8156029e>] ? do_page_fault+0x3e/0xa0
    Nov 27 12:53:23 localhost kernel: : [<ffffffff8155d265>] ? page_fault+0x25/0x30

    Nov 28 07:51:28 localhost kernel: : [32107] 8500 32107 2706 8 5 0 0 splunk
    Nov 28 07:51:28 localhost kernel: : [32831] 8500 32831 2706 28 3 0 0 splunk
    Nov 28 07:51:28 localhost kernel: : [32832] 8500 32832 27040 8 3 0 0 pid_check.sh
    Nov 28 07:51:28 localhost kernel: : [32833] 8500 32833 2706 7 0 0 0 splunk
    Nov 28 07:51:28 localhost kernel: : [32834] 8500 32834 27040 6 3 0 0 pid_check.sh

    Nov 28 07:51:28 localhost kernel: : Out of memory: Kill process 8787 (ossec-execd) score 1 or sacrifice child
    Nov 28 07:51:28 localhost kernel: : Killed process 8787, UID 0, (ossec-execd) total-vm:16952kB, anon-rss:0kB, file-rss:0kB
    Nov 28 07:51:28 localhost kernel: : pid_check.sh invoked oom-killer: gfp_mask=0xd0, order=2, oom_adj=0, oom_score_adj=0
    Nov 28 07:51:28 localhost kernel: : pid_check.sh cpuset=/ mems_allowed=0
    Nov 28 07:51:28 localhost kernel: : Pid: 32876, comm: pid_check.sh Not tainted 2.6.32-754.6.3.el6.x86_64 #1

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


In this Document
Symptoms
Cause
Solution
References


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.