How to reduce Page Allocation Failure events on Exadata environments. (Doc ID 1605814.1)

Last updated on JANUARY 03, 2017

Applies to:

Linux OS - Version Oracle Linux 5.0 to Oracle Linux 5.0 [Release OL5]
Oracle Exadata Storage Server Software - Version 11.2.1.3.0 to 11.2.3.2.1 [Release 11.2]
Oracle Exadata Storage Server Software - Version 11.2.3.3.0 to 11.2.3.3.0 [Release 11.2]
Oracle VM - Version 3.0.2 to 3.0.2 [Release OVM30]
Information in this document applies to any platform.

Symptoms

Node evictions are associated to different reasons.  But if Page Allocation Failure are a cause for node evictions, some facts are:

  1. process cssd will hit a Page Allocation Failure event. 
  2. It will not be any error recorded on the CRS log files.  
  3. Additionally,  no errors will be present in /var/log/messages
  4. OSW files will include data until few minutes before the reboot.
  5. System will report free memory (/proc/meminfo)


Some examples of the messages:

 

Nov  4 22:02:30 nodedb01 kernel: lowmem_reserve[]: 0 0 0 0
Nov  4 22:02:30 nodedb01 kernel: Node 0 DMA: 2*4kB 1*8kB 0*16kB 1*32kB 2*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15792kB
Nov  4 22:02:30 nodedb01 kernel: Node 0 DMA32: 4*4kB 7*8kB 6*16kB 8*32kB 7*64kB 5*128kB 2*256kB 7*512kB 7*1024kB 3*2048kB 103*4096kB = 440808kB
Nov  4 22:02:30 nodedb01 kernel: Node 0 Normal: 190505*4kB 32063*8kB 98*16kB 12*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 1*4096kB = 1026684kB

Nov  4 22:02:52 nodedb01 kernel: ocssd.bin: page allocation failure. order:2, mode:0x0
Nov  4 22:02:52 nodedb01 kernel: Pid: 13567, comm: ocssd.bin Tainted: P           2.6.32-400.29.1.el5uek #1
Nov  4 22:02:52 nodedb01 kernel: Call Trace:
Nov  4 22:02:52 nodedb01 kernel:  <IRQ>  [<ffffffff810de51b>] __alloc_pages_nodemask+0x524/0x595
Nov  4 22:02:52 nodedb01 kernel:  [<ffffffff8110e18b>] kmem_getpages+0x4f/0xf4
Nov  4 22:02:52 nodedb01 kernel:  [<ffffffff8110e388>] fallback_alloc+0x158/0x1ce
Nov  4 22:02:52 nodedb01 kernel:  [<ffffffff8110e51f>] ____cache_alloc_node+0x121/0x134
Nov  4 22:02:52 nodedb01 kernel:  [<ffffffff8110f8b8>] kmem_cache_alloc+0x7f/0xf7
Nov  4 22:02:52 nodedb01 kernel:  [<ffffffffa0383965>] rds_ib_recv_refill+0x154/0x524 [rds_rdma]
Nov  4 22:02:52 nodedb01 kernel:  [<ffffffffa0362a05>] ? rds_recv_incoming+0x222/0x22d [rds]
Nov  4 22:02:52 nodedb01 kernel:  [<ffffffffa0384350>] rds_ib_recv_cqe_handler+0x3fd/0x441 [rds_rdma]
Nov  4 22:02:52 nodedb01 kernel:  [<ffffffffa0382383>] poll_cq+0x4a/0x88 [rds_rdma]
Nov  4 22:02:52 nodedb01 kernel:  [<ffffffffa03821e0>] ? tasklet_schedule+0x1a/0x1c [rds_rdma]
Nov  4 22:02:53 nodedb01 kernel:  [<ffffffffa0382fa5>] rds_ib_tasklet_fn_recv+0x62/0xe6 [rds_rdma]
Nov  4 22:02:53 nodedb01 kernel:  [<ffffffff8105f0cd>] tasklet_action+0x79/0xe0
Nov  4 22:02:53 nodedb01 kernel:  [<ffffffff8105e8f9>] __do_softirq+0xd7/0x19e
Nov  4 22:02:53 nodedb01 kernel:  [<ffffffff810af580>] ? handle_IRQ_event+0x66/0x120
Nov  4 22:02:53 nodedb01 kernel:  [<ffffffff81012f2c>] call_softirq+0x1c/0x30
Nov  4 22:02:53 nodedb01 kernel:  [<ffffffff810146d5>] do_softirq+0x46/0x89
Nov  4 22:02:53 nodedb01 kernel:  [<ffffffff8105e77e>] irq_exit+0x3b/0x7a
Nov  4 22:02:53 nodedb01 kernel:  [<ffffffff8145d901>] do_IRQ+0x99/0xb0
Nov  4 22:02:53 nodedb01 kernel:  [<ffffffff81012753>] ret_from_intr+0x0/0x11

Nov  6 22:01:41 nodedb01 kernel: lowmem_reserve[]: 0 0 0 0
Nov  6 22:01:41 nodedb01 kernel: Node 0 DMA: 2*4kB 1*8kB 0*16kB 1*32kB 2*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15792kB
Nov  6 22:01:41 nodedb01 kernel: Node 0 DMA32: 4*4kB 7*8kB 6*16kB 8*32kB 7*64kB 5*128kB 2*256kB 7*512kB 7*1024kB 3*2048kB 103*4096kB = 440808kB
Nov  6 22:01:41 nodedb01 kernel: Node 0 Normal: 348289*4kB 509*8kB 70*16kB 2*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 1*4096kB = 1404556kB
Nov  6 22:01:41 nodedb01 kernel: 1994327 total pagecache pages
Nov  6 22:01:41 nodedb01 kernel: 201998 pages in swap cache
Nov  6 22:01:41 nodedb01 kernel: Swap cache stats: add 1166815, delete 964817, find 1919774/1982683
Nov  6 22:01:41 nodedb01 kernel: Free swap  = 23819560kB
Nov  6 22:01:41 nodedb01 kernel: Total swap = 25165816kB
Nov  6 22:01:41 nodedb01 kernel: ib0: transmit timeout: latency 12537 msecs
Nov  6 22:01:41 nodedb01 kernel: ib0: queue stopped 1, tx_head 1383628, tx_tail 1383628
Nov  6 22:01:41 nodedb01 kernel: ib0: transmit timeout: latency 13537 msecs
Nov  6 22:01:41 nodedb01 kernel: ib0: queue stopped 1, tx_head 1383628, tx_tail 1383628
Nov  6 22:01:41 nodedb01 kernel: 37748720 pages RAM
Nov  6 22:01:41 nodedb01 kernel: 572715 pages reserved
Nov  6 22:01:41 nodedb01 kernel: 387599424 pages shared
Nov  6 22:01:41 nodedb01 kernel: 10890781 pages non-shared
Nov  6 22:01:41 nodedb01 kernel: oracle: page allocation failure. order:1, mode:0x20
Nov  6 22:01:41 nodedb01 kernel: Pid: 28259, comm: oracle Tainted: P        W  2.6.32-400.29.1.el5uek #1
Nov  6 22:01:41 nodedb01 kernel: Call Trace:
Nov  6 22:01:41 nodedb01 kernel:  <IRQ>  [<ffffffff810de51b>] __alloc_pages_nodemask+0x524/0x595
Nov  6 22:01:41 nodedb01 kernel:  [<ffffffff8110e18b>] kmem_getpages+0x4f/0xf4
Nov  6 22:01:41 nodedb01 kernel:  [<ffffffff8110e388>] fallback_alloc+0x158/0x1ce
Nov  6 22:01:41 nodedb01 kernel:  [<ffffffff8110e51f>] ____cache_alloc_node+0x121/0x134
Nov  6 22:01:41 nodedb01 kernel:  [<ffffffff8110eb3f>] kmem_cache_alloc_node_notrace+0x84/0xb9
Nov  6 22:01:41 nodedb01 kernel:  [<ffffffff8110ebba>] __kmalloc_node+0x46/0x73
Nov  6 22:01:41 nodedb01 kernel:  [<ffffffff813bb4c8>] ? __alloc_skb+0x72/0x13d
Nov  6 22:01:41 nodedb01 kernel:  [<ffffffff813bb4c8>] __alloc_skb+0x72/0x13d
Nov  6 22:01:41 nodedb01 kernel:  [<ffffffff813bb5fb>] alloc_skb+0x13/0x15
Nov  6 22:01:41 nodedb01 kernel:  [<ffffffff813bb931>] dev_alloc_skb+0x1b/0x38
Nov  6 22:01:41 nodedb01 kernel:  [<ffffffffa034c722>] ipoib_cm_alloc_rx_skb+0x31/0x1de [ib_ipoib]
Nov  6 22:01:41 nodedb01 kernel:  [<ffffffffa034dd04>] ipoib_cm_handle_rx_wc+0x3a1/0x5b8 [ib_ipoib]
Nov  6 22:01:41 nodedb01 kernel:  [<ffffffffa024c4d4>] ? mlx4_ib_poll_cq+0x620/0x65e [mlx4_ib]

  


The Page Allocation Failure message include the amount of 4k contiguous pages that failed to be allocated (order:n).  Pages are requested in powers of 2 (2^n).
A node reporting this continuously is an indication of memory fragmentation, in particular when order is 0.

 

 

 # slabtop -o|grep rds
206186 206186 100%    8.00K 206186        1   3298976K rds_ib_incoming
206040 206040 100%    8.00K 206040        1   3296640K rds_ib_frag

  

 


 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms