Oracle VM: High Memory Consumption on Oracle VM Server, But Not Much Utilization by Processes (Doc ID 2471081.1)

Last updated on JUNE 01, 2023

Applies to:

Symptoms

An Oracle VM Server host version 3.4.4 or 3.4.5 host experiences high memory consumption, OOM kills processes, and possibly the host becomes hung. Backtraces similar to the one below might appear in /var/log/messages:

Sep 14 11:23:55 <hostname> kernel: [2267380.261629] python invoked oom-killer: gfp_mask=0x2000d0, order=2, oom_score_adj=0
Sep 14 11:23:55 <hostname> kernel: [2267380.261632] python cpuset=/ mems_allowed=0
Sep 14 11:23:55 <hostname> kernel: [2267380.261637] CPU: 8 PID: 28910 Comm: python Not tainted 4.1.12-124.17.1.el6uek.x86_64 #2
Sep 14 11:23:55 <hostname> kernel: [2267380.261638] Hardware name: Oracle Corporation ORACLE SERVER X7-2/ASM, MB, X7-2, BIOS 41021300 02/05/2018
Sep 14 11:23:55 <hostname> kernel: [2267380.261640] 0000000000000000 ffff8803978f3a88 ffffffff816e5323 ffff8803c8a07000
Sep 14 11:23:55 <hostname> kernel: [2267380.261642] 0000000000000000 ffff8803978f3b18 ffffffff816e3a19 ffffffff816f24e9
Sep 14 11:23:55 <hostname> kernel: [2267380.261644] ffffffff816f24e2 ffffffff816f24db ffffffff816f24d4 ffffffff816f24cd
Sep 14 11:23:55 <hostname> kernel: [2267380.261646] Call Trace:
Sep 14 11:23:55 <hostname> kernel: [2267380.261653] [<ffffffff816e5323>] dump_stack+0x63/0x81
Sep 14 11:23:55 <hostname> kernel: [2267380.261655] [<ffffffff816e3a19>] dump_header+0x7f/0x1f3
Sep 14 11:23:55 <hostname> kernel: [2267380.261658] [<ffffffff816f24e9>] ? xen_hypervisor_callback+0xa9/0x120
Sep 14 11:23:55 <hostname> kernel: [2267380.261660] [<ffffffff816f24e2>] ? xen_hypervisor_callback+0xa2/0x120
Sep 14 11:23:55 <hostname> kernel: [2267380.261662] [<ffffffff816f24db>] ? xen_hypervisor_callback+0x9b/0x120
Sep 14 11:23:55 <hostname> kernel: [2267380.261664] [<ffffffff816f24d4>] ? xen_hypervisor_callback+0x94/0x120
Sep 14 11:23:55 <hostname> kernel: [2267380.261665] [<ffffffff816f24cd>] ? xen_hypervisor_callback+0x8d/0x120
Sep 14 11:23:55 <hostname> kernel: [2267380.261667] [<ffffffff816f24c6>] ? xen_hypervisor_callback+0x86/0x120
Sep 14 11:23:55 <hostname> kernel: [2267380.261671] [<ffffffff8118e60c>] oom_kill_process+0x1cc/0x3c0
Sep 14 11:23:55 <hostname> kernel: [2267380.261673] [<ffffffff8118dfbd>] ? oom_unkillable_task+0xbd/0x100
Sep 14 11:23:55 <hostname> kernel: [2267380.261675] [<ffffffff8118eb7b>] __out_of_memory+0x31b/0x530
Sep 14 11:23:55 <hostname> kernel: [2267380.261677] [<ffffffff8118ef2b>] out_of_memory+0x5b/0x80
Sep 14 11:23:55 <hostname> kernel: [2267380.261680] [<ffffffff81194e52>] __alloc_pages_nodemask+0x952/0xab0
Sep 14 11:23:55 <hostname> kernel: [2267380.261683] [<ffffffff81139c19>] ? __delayacct_blkio_end+0x49/0x50
Sep 14 11:23:55 <hostname> kernel: [2267380.261686] [<ffffffff8119500f>] alloc_kmem_pages_node+0x5f/0x120
Sep 14 11:23:55 <hostname> kernel: [2267380.261690] [<ffffffff810820e1>] copy_process.part.26+0x111/0x19e0
Sep 14 11:23:55 <hostname> kernel: [2267380.261692] [<ffffffff81083b4c>] do_fork+0xbc/0x3a0
Sep 14 11:23:55 <hostname> kernel: [2267380.261695] [<ffffffff81026ddb>] ? syscall_trace_enter_phase1+0x11b/0x180
Sep 14 11:23:55 <hostname> kernel: [2267380.261697] [<ffffffff816edc26>] ? system_call_after_swapgs+0xf0/0x18e
Sep 14 11:23:55 <hostname> kernel: [2267380.261699] [<ffffffff816edc1f>] ? system_call_after_swapgs+0xe9/0x18e
Sep 14 11:23:55 <hostname> kernel: [2267380.261701] [<ffffffff81083eb6>] SyS_clone+0x16/0x20
Sep 14 11:23:55 <hostname> kernel: [2267380.261703] [<ffffffff816edcdc>] system_call_fastpath+0x18/0xd6
Sep 14 11:23:55 <hostname> kernel: [2267380.261704] Mem-Info:
Sep 14 11:23:55 <hostname> kernel: [2267380.261708] active_anon:0 inactive_anon:34 isolated_anon:0
Sep 14 11:23:55 <hostname> kernel: [2267380.261708] active_file:104 inactive_file:130 isolated_file:0
Sep 14 11:23:55 <hostname> kernel: [2267380.261708] unevictable:7598 dirty:0 writeback:0 unstable:0
Sep 14 11:23:55 <hostname> kernel: [2267380.261708] slab_reclaimable:21921 slab_unreclaimable:62974
Sep 14 11:23:55 <hostname> kernel: [2267380.261708] mapped:1118 shmem:0 pagetables:4118 bounce:0
Sep 14 11:23:55 <hostname> kernel: [2267380.261708] free:56402 free_pcp:0 free_cma:0
Sep 14 11:23:55 <hostname> kernel: [2267380.261712] Node 0 DMA free:14704kB min:76kB low:92kB high:108kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15988kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Sep 14 11:23:55 <hostname> kernel: [2267380.261717] lowmem_reserve[]: 0 1638 12899 12899
Sep 14 11:23:55 <hostname> kernel: [2267380.261719] Node 0 DMA32 free:100676kB min:8484kB low:10604kB high:12724kB active_anon:0kB inactive_anon:0kB active_file:164kB inactive_file:160kB unevictable:3852kB isolated(anon):0kB isolated(file):0kB present:1733184kB managed:1532084kB mlocked:3852kB dirty:0kB writeback:0kB mapped:360kB shmem:0kB slab_reclaimable:24436kB slab_unreclaimable:38988kB kernel_stack:5488kB pagetables:2436kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1996 all_unreclaimable? yes
Sep 14 11:23:55 <hostname> kernel: [2267380.261724] lowmem_reserve[]: 0 0 11261 11261
Sep 14 11:23:55 <hostname> kernel: [2267380.261726] Node 0 Normal free:110228kB min:56972kB low:71212kB high:85452kB active_anon:0kB inactive_anon:136kB active_file:252kB inactive_file:360kB unevictable:26540kB isolated(anon):0kB isolated(file):0kB present:11882312kB managed:10262036kB mlocked:26540kB dirty:0kB writeback:0kB mapped:4112kB shmem:0kB slab_reclaimable:63248kB slab_unreclaimable:212908kB kernel_stack:30880kB pagetables:14036kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:5524 all_unreclaimable? yes
Sep 14 11:23:55 <hostname> kernel: [2267380.261730] lowmem_reserve[]: 0 0 0 0
Sep 14 11:23:55 <hostname> kernel: [2267380.261732] Node 0 DMA: 0*4kB 0*8kB 1*16kB (U) 1*32kB (U) 3*64kB (U) 1*128kB (U) 0*256kB 0*512kB 0*1024kB 1*2048kB (R) 3*4096kB (M) = 14704kB
Sep 14 11:23:55 <hostname> kernel: [2267380.261740] Node 0 DMA32: 17076*4kB (UEM) 3802*8kB (UEM) 1*16kB (U) 4*32kB (R) 7*64kB (R) 4*128kB (R) 2*256kB (R) 1*512kB (R) 0*1024kB 0*2048kB 0*4096kB = 100848kB
Sep 14 11:23:55 <hostname> kernel: [2267380.261748] Node 0 Normal: 12044*4kB (UEM) 6169*8kB (UEM) 610*16kB (UEM) 1*32kB (R) 2*64kB (R) 0*128kB 1*256kB (R) 1*512kB (R) 1*1024kB (R) 1*2048kB (R) 0*4096kB = 111288kB
Sep 14 11:23:55 <hostname> kernel: [2267380.261756] 1401 total pagecache pages
Sep 14 11:23:55 <hostname> kernel: [2267380.261757] 32 pages in swap cache
Sep 14 11:23:55 <hostname> kernel: [2267380.261759] Swap cache stats: add 3847620, delete 3847588, find 58507020/60346214
Sep 14 11:23:55 <hostname> kernel: [2267380.261760] Free swap = 3759288kB
Sep 14 11:23:55 <hostname> kernel: [2267380.261761] Total swap = 4194300kB
Sep 14 11:23:55 <hostname> kernel: [2267380.261761] 3407871 pages RAM
Sep 14 11:23:55 <hostname> kernel: [2267380.261762] 0 pages HighMem/MovableOnly
Sep 14 11:23:55 <hostname> kernel: [2267380.261763] 455366 pages reserved
Sep 14 11:23:55 <hostname> kernel: [2267380.261764] 0 pages cma reserved
Sep 14 11:23:55 <hostname> kernel: [2267380.261765] 0 pages hwpoisoned
Sep 14 11:23:55 <hostname> kernel: [2267380.261765] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
Sep 14 11:23:55 <hostname> kernel: [2267380.261789] [ 7756] 0 7756 2677 213 10 3 105 -1000 udevd
Sep 14 11:23:55 <hostname> kernel: [2267380.261870] [22812] 0 22812 2620766 5899 209 23 0 -1000 multipathd
Sep 14 11:23:55 <hostname> kernel: [2267380.261889] [27791] 0 27791 22785 325 15 3 82 -1000 auditd
Sep 14 11:23:55 <hostname> kernel: [2267380.261891] [27825] 0 27825 61275 302 21 4 804 0 rsyslogd
Sep 14 11:23:55 <hostname> kernel: [2267380.261893] [27840] 0 27840 3293 362 12 3 614

Changes

This can happen with fresh installation or after an upgrade.

Cause

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.