DBFS :: High Memory usage by DBFS_Client leading to node evcition

(Doc ID 2317570.1)

Last updated on APRIL 13, 2018

Applies to:

Oracle Database - Enterprise Edition - Version 12.1.0.2 and later
Information in this document applies to any platform.

Symptoms


With XAG installation on DBFS filesystem which we use to fail Goldengate between two nodes of the 12c RAC.
System level free memory went so low and OS level OOM killer was invoked .

 

zzz ***Sat Mar 4 00:46:04 GMT 2017
MemTotal: 132118812 kB
MemFree: 352580 kB <<-----
Buffers: 3748 kB
Cached: 235812 kB
SwapCached: 32440 kB
Active: 58356168 kB
Inactive: 3966332 kB
Active(anon): 58343608 kB

File_name:: xxxxxx.xxxx.xxx.xxx.com_meminfo_17.03.04.0000.dat

+ System was running on low free memory

 

kswapd was called in using comparatively higher CPUs.
Mar 4 00:45:28 xxxxxx kernel: Out of memory: Kill process 52347 (dbfs_client) score 637 or sacrifice child
Mar 4 00:45:28 xxxxxx kernel: Killed process 55393, UID 25265, (oracle_55393_cm) total-vm:26008092kB, anon-rss:10232kB, file-rss:1688kB
Mar 4 00:45:28 xxxxxx kernel: fglam invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Mar 4 00:45:28 xxxxxx kernel: fglam cpuset=/ mems_allowed=0-1
Mar 4 00:45:28 xxxxxx kernel: Pid: 11705, comm: fglam Tainted: P -- ------------ 2.6.32-642.11.1.el6.x86_64 #1
Mar 4 00:45:28 xxxxxx kernel: Call Trace:
Mar 4 00:45:28 xxxxxx kernel: [] ? dump_header+0x90/0x1b0
Mar 4 00:45:28 xxxxxx kernel: [] ? cpuset_mems_allowed_intersects+0x21/0x30
Mar 4 00:45:28 xxxxxx kernel: [] ? oom_kill_process+0x82/0x2a0
Mar 4 00:45:28 xxxxxx kernel: [] ? select_bad_process+0xe1/0x120
Mar 4 00:45:28 xxxxxx kernel: [] ? out_of_memory+0x220/0x3c0
Mar 4 00:45:28 xxxxxx kernel: [] ? __alloc_pages_nodemask+0x93c/0x950
Mar 4 00:45:28 xxxxxx kernel: [] ? alloc_pages_current+0xaa/0x110
Mar 4 00:45:28 xxxxxx kernel: [] ? __page_cache_alloc+0x87/0x90
Mar 4 00:45:28 xxxxxx kernel: [] ? find_get_page+0x1e/0xa0
Mar 4 00:45:28 xxxxxx kernel: [] ? filemap_fault+0x1a7/0x500
Mar 4 00:45:28 xxxxxx kernel: [] ? __do_fault+0x54/0x530
Mar 4 00:45:28 xxxxxx kernel: [] ? handle_pte_fault+0xf7/0xb20
Mar 4 00:45:28 xxxxxx kernel: [] ? handle_mm_fault+0x299/0x3d0
Mar 4 00:45:28 xxxxxx kernel: [] ? __do_page_fault+0x146/0x500
Mar 4 00:45:28 xxxxxx kernel: [] ? memzero_explicit+0xe/0x10
Mar 4 00:45:28 xxxxxx kernel: [] ? extract_entropy_user+0xd8/0x130
Mar 4 00:45:28 xxxxxx kernel: [] ? show_registers+0xd0/0x280
Mar 4 00:45:28 xxxxxx kernel: [] ? do_page_fault+0x3e/0xa0
Mar 4 00:45:28 xxxxxx kernel: [] ? page_fault+0x25/0x30
Mar 4 00:45:28 xxxxxx kernel: Mem-Info:
Mar 4 00:45:28 xxxxxx kernel: Node 0 DMA per-cpu:

Mar 4 00:46:28 xxxxxx kernel: Out of memory: Kill process 52347 (dbfs_client) score 637 or sacrifice child
Mar 4 00:46:28 xxxxxx kernel: Killed process 5162, UID 25265, (oracle_5162_cmp) total-vm:26007964kB, anon-rss:8336kB, file-rss:1444kB
Mar 4 00:46:28 xxxxxx kernel: enlinuxpc64 invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Mar 4 00:46:28 xxxxxx kernel: enlinuxpc64 cpuset=/ mems_allowed=0-1
Mar 4 00:46:28 xxxxxx kernel: Pid: 9096, comm: enlinuxpc64 Tainted: P -- ------------ 2.6.32-642.11.1.el6.x86_64 #1
Mar 4 00:46:28 xxxxxx kernel: Call Trace:
Mar 4 00:46:28 xxxxxx kernel: [] ? dump_header+0x90/0x1b0
Mar 4 00:46:28 xxxxxx kernel: [] ? cpuset_mems_allowed_intersects+0x21/0x30
Mar 4 00:46:28 xxxxxx kernel: [] ? oom_kill_process+0x82/0x2a0
Mar 4 00:46:28 xxxxxx kernel: [] ? select_bad_process+0xe1/0x120
Mar 4 00:46:28 xxxxxx kernel: [] ? out_of_memory+0x220/0x3c0
Mar 4 00:46:28 xxxxxx kernel: [] ? __alloc_pages_nodemask+0x93c/0x950
Mar 4 00:46:28 xxxxxx kernel: [] ? alloc_pages_current+0xaa/0x110
Mar 4 00:46:28 xxxxxx kernel: [] ? __page_cache_alloc+0x87/0x90
Mar 4 00:46:28 xxxxxx kernel: [] ? find_get_page+0x1e/0xa0
Mar 4 00:46:28 xxxxxx kernel: [] ? filemap_fault+0x1a7/0x500
Mar 4 00:46:28 xxxxxx kernel: [] ? __do_fault+0x54/0x530
Mar 4 00:46:28 xxxxxx kernel: [] ? handle_pte_fault+0xf7/0xb20
Mar 4 00:46:28 xxxxxx kernel: [] ? apic_timer_interrupt+0xe/0x20
Mar 4 00:46:28 xxxxxx kernel: [] ? call_function_interrupt+0xe/0x20
Mar 4 00:46:28 xxxxxx kernel: [] ? copy_user_generic+0x9/0x10
Mar 4 00:46:28 xxxxxx kernel: [] ? handle_mm_fault+0x299/0x3d0
Mar 4 00:46:28 xxxxxx kernel: [] ? __do_page_fault+0x146/0x500
Mar 4 00:46:28 xxxxxx kernel: [] ? read_tsc+0x9/0x10
Mar 4 00:46:28 xxxxxx kernel: [] ? ktime_get_ts+0xbf/0x100
Mar 4 00:46:28 xxxxxx kernel: [] ? poll_select_copy_remaining+0xf8/0x150
Mar 4 00:46:28 xxxxxx kernel: [] ? do_page_fault+0x3e/0xa0
Mar 4 00:46:28 xxxxxx kernel: [] ? page_fault+0x25/0x30
Mar 4 00:46:28 xxxxxx kernel: Mem-Info:
Mar 4 00:46:28 xxxxxx kernel: Node 0 DMA per-cpu:
Mar 4

File_name :: messages

OS level OOM was called in kill OS level processes

Line 12: 52347 oracle 20 0 137g 53g 4388 S 52.4 42.3 2232:37 dbfs_client
Line 62: 52347 oracle 20 0 137g 53g 4388 S 28.1 42.3 2232:52 dbfs_client
Line 115: 52347 oracle 20 0 137g 53g 4392 S 48.4 42.3 2233:07 dbfs_client
Line 166: 52347 oracle 20 0 137g 53g 4392 S 52.3 42.3 2233:23 dbfs_client
Line 217: 52347 oracle 20 0 137g 53g 4388 S 51.3 42.3 2233:39 dbfs_client
Line 267: 52347 oracle 20 0 137g 53g 4300 S 51.3 42.4 2233:53 dbfs_client
Line 317: 52347 oracle 20 0 137g 53g 4308 S 54.2 42.4 2234:09 dbfs_client

Line 4246: 52347 oracle 20 0 139g 54g 4220 S 55.7 42.9 2254:10 dbfs_client
Line 4298: 52347 oracle 20 0 139g 54g 3648 S 8.7 42.9 2254:24 dbfs_client
Line 4432: 52347 oracle 20 0 139g 54g 2952 S 1.0 42.9 2254:36 dbfs_client
Line 4451: 52347 oracle 20 0 139g 54g 3320 S 5.7 42.9 2254:44 dbfs_client
Line 4601: 52347 oracle 20 0 139g 54g 4512 S 72.6 42.9 2254:51 dbfs_client
Line 4908: 18800 oracle 20 0 1896m 217m 11m S 15.5 0.2 0:01.23 dbfs_client
Line 5015: 18800 oracle 20 0 1896m 149m 11m S 6.8 0.1 0:03.32 dbfs_client
Line 5059: 18800 oracle 20 0 2886m 76m 11m S 15.5 0.1 0:07.25 dbfs_client

File_name :: xxxxxx.xxxx.xxx.xxx.com_top_17.03.04.0200.dat

 

 

dbfs_client process memory consumption was increasing monotonically.

Changes

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms