How to reduce Page Allocation Failure events on Exadata environments.
(Doc ID 1605814.1)
Last updated on OCTOBER 09, 2024
Applies to:
Oracle Exadata Storage Server Software - Version 11.2.1.3.0 to 11.2.3.2.1 [Release 11.2]Oracle Exadata Storage Server Software - Version 11.2.3.3.0 to 11.2.3.3.0 [Release 11.2]
Oracle VM - Version 3.0.2 to 3.0.2 [Release OVM30]
Linux OS - Version Oracle Linux 5.0 to Oracle Linux 5.0 [Release OL5]
Oracle Cloud Infrastructure - Version N/A and later
Information in this document applies to any platform.
Symptoms
- Node evictions
Node evictions are associated to different reasons. But if Page Allocation Failure are a cause for node evictions, some facts are:
- process cssd will hit a Page Allocation Failure event.
- It will not be any error recorded on the CRS log files.
- Additionally, no errors will be present in /var/log/messages
- OSW files will include data until few minutes before the reboot.
- System will report free memory (/proc/meminfo)
- Page Allocation Failures reported on /var/log/messages file.
Some examples of the messages:
Nov 4 22:02:30 nodedb01 kernel: lowmem_reserve[]: 0 0 0 0
Nov 4 22:02:30 nodedb01 kernel: Node 0 DMA: 2*4kB 1*8kB 0*16kB 1*32kB 2*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15792kB
Nov 4 22:02:30 nodedb01 kernel: Node 0 DMA32: 4*4kB 7*8kB 6*16kB 8*32kB 7*64kB 5*128kB 2*256kB 7*512kB 7*1024kB 3*2048kB 103*4096kB = 440808kB
Nov 4 22:02:30 nodedb01 kernel: Node 0 Normal: 190505*4kB 32063*8kB 98*16kB 12*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 1*4096kB = 1026684kB
Nov 4 22:02:52 nodedb01 kernel: ocssd.bin: page allocation failure. order:2, mode:0x0
Nov 4 22:02:52 nodedb01 kernel: Pid: 13567, comm: ocssd.bin Tainted: P 2.6.32-400.29.1.el5uek #1
Nov 4 22:02:52 nodedb01 kernel: Call Trace:
Nov 4 22:02:52 nodedb01 kernel: <IRQ> [<ffffffff810de51b>] __alloc_pages_nodemask+0x524/0x595
Nov 4 22:02:52 nodedb01 kernel: [<ffffffff8110e18b>] kmem_getpages+0x4f/0xf4
Nov 4 22:02:52 nodedb01 kernel: [<ffffffff8110e388>] fallback_alloc+0x158/0x1ce
Nov 4 22:02:52 nodedb01 kernel: [<ffffffff8110e51f>] ____cache_alloc_node+0x121/0x134
Nov 4 22:02:52 nodedb01 kernel: [<ffffffff8110f8b8>] kmem_cache_alloc+0x7f/0xf7
Nov 4 22:02:52 nodedb01 kernel: [<ffffffffa0383965>] rds_ib_recv_refill+0x154/0x524 [rds_rdma]
Nov 4 22:02:52 nodedb01 kernel: [<ffffffffa0362a05>] ? rds_recv_incoming+0x222/0x22d [rds]
Nov 4 22:02:52 nodedb01 kernel: [<ffffffffa0384350>] rds_ib_recv_cqe_handler+0x3fd/0x441 [rds_rdma]
Nov 4 22:02:52 nodedb01 kernel: [<ffffffffa0382383>] poll_cq+0x4a/0x88 [rds_rdma]
Nov 4 22:02:52 nodedb01 kernel: [<ffffffffa03821e0>] ? tasklet_schedule+0x1a/0x1c [rds_rdma]
Nov 4 22:02:53 nodedb01 kernel: [<ffffffffa0382fa5>] rds_ib_tasklet_fn_recv+0x62/0xe6 [rds_rdma]
Nov 4 22:02:53 nodedb01 kernel: [<ffffffff8105f0cd>] tasklet_action+0x79/0xe0
Nov 4 22:02:53 nodedb01 kernel: [<ffffffff8105e8f9>] __do_softirq+0xd7/0x19e
Nov 4 22:02:53 nodedb01 kernel: [<ffffffff810af580>] ? handle_IRQ_event+0x66/0x120
Nov 4 22:02:53 nodedb01 kernel: [<ffffffff81012f2c>] call_softirq+0x1c/0x30
Nov 4 22:02:53 nodedb01 kernel: [<ffffffff810146d5>] do_softirq+0x46/0x89
Nov 4 22:02:53 nodedb01 kernel: [<ffffffff8105e77e>] irq_exit+0x3b/0x7a
Nov 4 22:02:53 nodedb01 kernel: [<ffffffff8145d901>] do_IRQ+0x99/0xb0
Nov 4 22:02:53 nodedb01 kernel: [<ffffffff81012753>] ret_from_intr+0x0/0x11
Nov 6 22:01:41 nodedb01 kernel: lowmem_reserve[]: 0 0 0 0
Nov 6 22:01:41 nodedb01 kernel: Node 0 DMA: 2*4kB 1*8kB 0*16kB 1*32kB 2*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15792kB
Nov 6 22:01:41 nodedb01 kernel: Node 0 DMA32: 4*4kB 7*8kB 6*16kB 8*32kB 7*64kB 5*128kB 2*256kB 7*512kB 7*1024kB 3*2048kB 103*4096kB = 440808kB
Nov 6 22:01:41 nodedb01 kernel: Node 0 Normal: 348289*4kB 509*8kB 70*16kB 2*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 1*4096kB = 1404556kB
Nov 6 22:01:41 nodedb01 kernel: 1994327 total pagecache pages
Nov 6 22:01:41 nodedb01 kernel: 201998 pages in swap cache
Nov 6 22:01:41 nodedb01 kernel: Swap cache stats: add 1166815, delete 964817, find 1919774/1982683
Nov 6 22:01:41 nodedb01 kernel: Free swap = 23819560kB
Nov 6 22:01:41 nodedb01 kernel: Total swap = 25165816kB
Nov 6 22:01:41 nodedb01 kernel: ib0: transmit timeout: latency 12537 msecs
Nov 6 22:01:41 nodedb01 kernel: ib0: queue stopped 1, tx_head 1383628, tx_tail 1383628
Nov 6 22:01:41 nodedb01 kernel: ib0: transmit timeout: latency 13537 msecs
Nov 6 22:01:41 nodedb01 kernel: ib0: queue stopped 1, tx_head 1383628, tx_tail 1383628
Nov 6 22:01:41 nodedb01 kernel: 37748720 pages RAM
Nov 6 22:01:41 nodedb01 kernel: 572715 pages reserved
Nov 6 22:01:41 nodedb01 kernel: 387599424 pages shared
Nov 6 22:01:41 nodedb01 kernel: 10890781 pages non-shared
Nov 6 22:01:41 nodedb01 kernel: oracle: page allocation failure. order:1, mode:0x20
Nov 6 22:01:41 nodedb01 kernel: Pid: 28259, comm: oracle Tainted: P W 2.6.32-400.29.1.el5uek #1
Nov 6 22:01:41 nodedb01 kernel: Call Trace:
Nov 6 22:01:41 nodedb01 kernel: <IRQ> [<ffffffff810de51b>] __alloc_pages_nodemask+0x524/0x595
Nov 6 22:01:41 nodedb01 kernel: [<ffffffff8110e18b>] kmem_getpages+0x4f/0xf4
Nov 6 22:01:41 nodedb01 kernel: [<ffffffff8110e388>] fallback_alloc+0x158/0x1ce
Nov 6 22:01:41 nodedb01 kernel: [<ffffffff8110e51f>] ____cache_alloc_node+0x121/0x134
Nov 6 22:01:41 nodedb01 kernel: [<ffffffff8110eb3f>] kmem_cache_alloc_node_notrace+0x84/0xb9
Nov 6 22:01:41 nodedb01 kernel: [<ffffffff8110ebba>] __kmalloc_node+0x46/0x73
Nov 6 22:01:41 nodedb01 kernel: [<ffffffff813bb4c8>] ? __alloc_skb+0x72/0x13d
Nov 6 22:01:41 nodedb01 kernel: [<ffffffff813bb4c8>] __alloc_skb+0x72/0x13d
Nov 6 22:01:41 nodedb01 kernel: [<ffffffff813bb5fb>] alloc_skb+0x13/0x15
Nov 6 22:01:41 nodedb01 kernel: [<ffffffff813bb931>] dev_alloc_skb+0x1b/0x38
Nov 6 22:01:41 nodedb01 kernel: [<ffffffffa034c722>] ipoib_cm_alloc_rx_skb+0x31/0x1de [ib_ipoib]
Nov 6 22:01:41 nodedb01 kernel: [<ffffffffa034dd04>] ipoib_cm_handle_rx_wc+0x3a1/0x5b8 [ib_ipoib]
Nov 6 22:01:41 nodedb01 kernel: [<ffffffffa024c4d4>] ? mlx4_ib_poll_cq+0x620/0x65e [mlx4_ib]
Nov 4 22:02:30 nodedb01 kernel: Node 0 DMA: 2*4kB 1*8kB 0*16kB 1*32kB 2*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15792kB
Nov 4 22:02:30 nodedb01 kernel: Node 0 DMA32: 4*4kB 7*8kB 6*16kB 8*32kB 7*64kB 5*128kB 2*256kB 7*512kB 7*1024kB 3*2048kB 103*4096kB = 440808kB
Nov 4 22:02:30 nodedb01 kernel: Node 0 Normal: 190505*4kB 32063*8kB 98*16kB 12*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 1*4096kB = 1026684kB
Nov 4 22:02:52 nodedb01 kernel: ocssd.bin: page allocation failure. order:2, mode:0x0
Nov 4 22:02:52 nodedb01 kernel: Pid: 13567, comm: ocssd.bin Tainted: P 2.6.32-400.29.1.el5uek #1
Nov 4 22:02:52 nodedb01 kernel: Call Trace:
Nov 4 22:02:52 nodedb01 kernel: <IRQ> [<ffffffff810de51b>] __alloc_pages_nodemask+0x524/0x595
Nov 4 22:02:52 nodedb01 kernel: [<ffffffff8110e18b>] kmem_getpages+0x4f/0xf4
Nov 4 22:02:52 nodedb01 kernel: [<ffffffff8110e388>] fallback_alloc+0x158/0x1ce
Nov 4 22:02:52 nodedb01 kernel: [<ffffffff8110e51f>] ____cache_alloc_node+0x121/0x134
Nov 4 22:02:52 nodedb01 kernel: [<ffffffff8110f8b8>] kmem_cache_alloc+0x7f/0xf7
Nov 4 22:02:52 nodedb01 kernel: [<ffffffffa0383965>] rds_ib_recv_refill+0x154/0x524 [rds_rdma]
Nov 4 22:02:52 nodedb01 kernel: [<ffffffffa0362a05>] ? rds_recv_incoming+0x222/0x22d [rds]
Nov 4 22:02:52 nodedb01 kernel: [<ffffffffa0384350>] rds_ib_recv_cqe_handler+0x3fd/0x441 [rds_rdma]
Nov 4 22:02:52 nodedb01 kernel: [<ffffffffa0382383>] poll_cq+0x4a/0x88 [rds_rdma]
Nov 4 22:02:52 nodedb01 kernel: [<ffffffffa03821e0>] ? tasklet_schedule+0x1a/0x1c [rds_rdma]
Nov 4 22:02:53 nodedb01 kernel: [<ffffffffa0382fa5>] rds_ib_tasklet_fn_recv+0x62/0xe6 [rds_rdma]
Nov 4 22:02:53 nodedb01 kernel: [<ffffffff8105f0cd>] tasklet_action+0x79/0xe0
Nov 4 22:02:53 nodedb01 kernel: [<ffffffff8105e8f9>] __do_softirq+0xd7/0x19e
Nov 4 22:02:53 nodedb01 kernel: [<ffffffff810af580>] ? handle_IRQ_event+0x66/0x120
Nov 4 22:02:53 nodedb01 kernel: [<ffffffff81012f2c>] call_softirq+0x1c/0x30
Nov 4 22:02:53 nodedb01 kernel: [<ffffffff810146d5>] do_softirq+0x46/0x89
Nov 4 22:02:53 nodedb01 kernel: [<ffffffff8105e77e>] irq_exit+0x3b/0x7a
Nov 4 22:02:53 nodedb01 kernel: [<ffffffff8145d901>] do_IRQ+0x99/0xb0
Nov 4 22:02:53 nodedb01 kernel: [<ffffffff81012753>] ret_from_intr+0x0/0x11
Nov 6 22:01:41 nodedb01 kernel: lowmem_reserve[]: 0 0 0 0
Nov 6 22:01:41 nodedb01 kernel: Node 0 DMA: 2*4kB 1*8kB 0*16kB 1*32kB 2*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15792kB
Nov 6 22:01:41 nodedb01 kernel: Node 0 DMA32: 4*4kB 7*8kB 6*16kB 8*32kB 7*64kB 5*128kB 2*256kB 7*512kB 7*1024kB 3*2048kB 103*4096kB = 440808kB
Nov 6 22:01:41 nodedb01 kernel: Node 0 Normal: 348289*4kB 509*8kB 70*16kB 2*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 1*4096kB = 1404556kB
Nov 6 22:01:41 nodedb01 kernel: 1994327 total pagecache pages
Nov 6 22:01:41 nodedb01 kernel: 201998 pages in swap cache
Nov 6 22:01:41 nodedb01 kernel: Swap cache stats: add 1166815, delete 964817, find 1919774/1982683
Nov 6 22:01:41 nodedb01 kernel: Free swap = 23819560kB
Nov 6 22:01:41 nodedb01 kernel: Total swap = 25165816kB
Nov 6 22:01:41 nodedb01 kernel: ib0: transmit timeout: latency 12537 msecs
Nov 6 22:01:41 nodedb01 kernel: ib0: queue stopped 1, tx_head 1383628, tx_tail 1383628
Nov 6 22:01:41 nodedb01 kernel: ib0: transmit timeout: latency 13537 msecs
Nov 6 22:01:41 nodedb01 kernel: ib0: queue stopped 1, tx_head 1383628, tx_tail 1383628
Nov 6 22:01:41 nodedb01 kernel: 37748720 pages RAM
Nov 6 22:01:41 nodedb01 kernel: 572715 pages reserved
Nov 6 22:01:41 nodedb01 kernel: 387599424 pages shared
Nov 6 22:01:41 nodedb01 kernel: 10890781 pages non-shared
Nov 6 22:01:41 nodedb01 kernel: oracle: page allocation failure. order:1, mode:0x20
Nov 6 22:01:41 nodedb01 kernel: Pid: 28259, comm: oracle Tainted: P W 2.6.32-400.29.1.el5uek #1
Nov 6 22:01:41 nodedb01 kernel: Call Trace:
Nov 6 22:01:41 nodedb01 kernel: <IRQ> [<ffffffff810de51b>] __alloc_pages_nodemask+0x524/0x595
Nov 6 22:01:41 nodedb01 kernel: [<ffffffff8110e18b>] kmem_getpages+0x4f/0xf4
Nov 6 22:01:41 nodedb01 kernel: [<ffffffff8110e388>] fallback_alloc+0x158/0x1ce
Nov 6 22:01:41 nodedb01 kernel: [<ffffffff8110e51f>] ____cache_alloc_node+0x121/0x134
Nov 6 22:01:41 nodedb01 kernel: [<ffffffff8110eb3f>] kmem_cache_alloc_node_notrace+0x84/0xb9
Nov 6 22:01:41 nodedb01 kernel: [<ffffffff8110ebba>] __kmalloc_node+0x46/0x73
Nov 6 22:01:41 nodedb01 kernel: [<ffffffff813bb4c8>] ? __alloc_skb+0x72/0x13d
Nov 6 22:01:41 nodedb01 kernel: [<ffffffff813bb4c8>] __alloc_skb+0x72/0x13d
Nov 6 22:01:41 nodedb01 kernel: [<ffffffff813bb5fb>] alloc_skb+0x13/0x15
Nov 6 22:01:41 nodedb01 kernel: [<ffffffff813bb931>] dev_alloc_skb+0x1b/0x38
Nov 6 22:01:41 nodedb01 kernel: [<ffffffffa034c722>] ipoib_cm_alloc_rx_skb+0x31/0x1de [ib_ipoib]
Nov 6 22:01:41 nodedb01 kernel: [<ffffffffa034dd04>] ipoib_cm_handle_rx_wc+0x3a1/0x5b8 [ib_ipoib]
Nov 6 22:01:41 nodedb01 kernel: [<ffffffffa024c4d4>] ? mlx4_ib_poll_cq+0x620/0x65e [mlx4_ib]
The Page Allocation Failure message include the amount of 4k contiguous pages that failed to be allocated (order:n). Pages are requested in powers of 2 (2^n).
A node reporting this continuously is an indication of memory fragmentation, in particular when order is 0.
- It may be possible that RDS is one of the largest contributor to the cache allocation, leading to memory fragmentation. Values beyond 1GB and increasing is a red flag.
# slabtop -o|grep rds
206186 206186 100% 8.00K 206186 1 3298976K rds_ib_incoming
206040 206040 100% 8.00K 206040 1 3296640K rds_ib_frag
Cause
To view full details, sign in with your My Oracle Support account. |
|
Don't have a My Oracle Support account? Click to get started! |