OSB - Using RDS / RDMA over InfiniBand
(Doc ID 1510603.1)
Last updated on DECEMBER 29, 2017
Applies to:Oracle Secure Backup - Version 10.4.0.1 to 10.4.0.1 [Release 10.4]
Oracle Solaris on SPARC (64-bit)
Oracle Solaris on x86-64 (64-bit)
InfiniBand is hardware interconnect that provides a very high bandwidth and low latency. Reliable Datagram Socket (RDS) is an open source protocol developed by Oracle and partners to be used over Infiniband. The RDS protocol offloads error checking to the hardware thereby keeping the CPU utilization low and is therefore preferred over using Internet Protocol over InfiniBand (IPoIB).
Oracle Secure Backup (OSB) transfers data over InfiniBand using the Reliable Datagram Socket (RDS) protocol as supported by the host and operating system. If RDS has not been installed on the host or is not available, OSB will use TCP/IP over IB. By default, OSB will preference RDS over TCP/IP in IB environments. With the addition of a zero-copy extension , Remote Direct Memory Access (RDMA), the advantages of RDS were further increased. Oracle Secure Backup leverages both RDS / RDMA over IB as supported in the environment.
Setup of InfiniBand and RDS / RDMA is outside the scope of OSB. However, it must be configured properly in order for successful communication to occur. OSB does NOT have an explicit certification matrix detailing specific operating system or kernel versions etc specific to InfiniBand / RDS / RDMA enviroments. OSB must support the host platform and operating system as is always the case. If the IB network between two hosts is operational and can communicate via RDS / RDMA, then the environment is presumably configured properly and OSB will be able to leverage RDS / RDMA over IB.
A few tips for consideration when configuring / testing the networking infrastructure:
After hardware configuration, insure the following steps have been performed (machine reboot is required after these changes are made:
- Configure the correct MTU size and connection mode for your system
- On Linux run “chkconfig openibd on” and on Solaris run "svcadm enable svc:/system/rds" – this will make sure that every time the machine is rebooted all the RDS/RDMA modules are loaded
- If using Linux, modify the rmem_default, if required to prevent ENOBUFs, make sure this isn’t greater than rmem_max setting – echo 4194304 > /proc/sys/net/core/rmem_default
Good to know checks when using Infiniband/RDS/RDMA:
- Rds-ping – this can be used to verify RDS/IB connectivity. - Note: You should be able to use rds-ping to ping local IB interface via RDS even if there isn’t another remote machine available. For e.g on host55, “rds-ping –c –I 192.168.31.220 192.168.31.220” should work if everything is configured correctly.
- Rds-stress – this can be run between two machines connected via Infiniband to assess RDS/RDMA performance
- Ibstat – this can be used to check the status of IB interfaces on a host
- If using Linux, check the MTU setting of an IB interface – “cat /sys/class/net/ib0/mtu”
- If using Linux, check the connectivity setting of an IB interface – “cat /sys/class/net/ib0/mode” – should be set to “connected”
Refer to the following documentation for Monitoring and Troubleshooting IB devices: http://docs.oracle.com/cd/E23824_01/html/821-1459/gjwwf.html
While the documentation link is for Solaris, it provides general RDS commands which are applicable to other operating systems.
To view full details, sign in with your My Oracle Support account.
Don't have a My Oracle Support account? Click to get started!