OSB - Using RDS / RDMA over InfiniBand (Doc ID 1510603.1)

Last updated on SEPTEMBER 16, 2015

Applies to:

Oracle Secure Backup - Version 10.4.0.1 to 10.4.0.1 [Release 10.4]
Linux x86-64
Oracle Solaris on SPARC (64-bit)
Oracle Solaris on x86-64 (64-bit)
InfiniBand is hardware interconnect that provides a very high bandwidth and low latency. Reliable Datagram Socket (RDS) is an open source protocol developed by Oracle and partners to be used over Infiniband. The RDS protocol offloads error checking to the hardware thereby keeping the CPU utilization low and is therefore preferred over using Internet Protocol over InfiniBand (IPoIB).

Oracle Secure Backup (OSB) transfers data over InfiniBand using the Reliable Datagram Socket (RDS) protocol as supported by the host and operating system. If RDS has not been installed on the host or is not available, OSB will use TCP/IP over IB. By default, OSB will preference RDS over TCP/IP in IB environments. With the addition of a zero-copy extension , Remote Direct Memory Access (RDMA), the advantages of RDS were further increased. Oracle Secure Backup leverages both RDS / RDMA over IB as supported in the environment.

Goal

Setup of InfiniBand and RDS / RDMA is outside the scope of OSB.  However, it must be configured properly in order for successful communication to occur.  OSB does NOT have an explicit certification matrix detailing specific operating system or kernel versions etc specific to InfiniBand / RDS / RDMA enviroments.     OSB must support the host platform and operating system as is always the case.   If the IB network between two hosts is operational and can communicate via RDS / RDMA, then the environment is presumably configured properly and OSB will be able to leverage RDS / RDMA over IB.

A few tips for consideration when configuring / testing the networking infrastructure:

After hardware configuration, insure the following steps have been performed (machine reboot is required after these changes are made: 

  1. Configure the correct MTU size and connection mode for your system
  2. On Linux run “chkconfig openibd on” and on Solaris run "svcadm enable svc:/system/rds"  – this will make sure that every time the machine is rebooted all the RDS/RDMA modules are loaded
  3. If using Linux, modify the rmem_default, if required to prevent ENOBUFs, make sure this isn’t greater than rmem_max setting –  echo 4194304 > /proc/sys/net/core/rmem_default

 Good to know checks when using Infiniband/RDS/RDMA:

 Refer to the following documentation for Monitoring and Troubleshooting IB devices:  http://docs.oracle.com/cd/E23824_01/html/821-1459/gjwwf.html

 While the documentation link is for Solaris, it provides general RDS commands which are applicable to other operating systems.

Solution

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms