Link Aggregation Control Protocol (LACP) on Exadata (Doc ID 2198475.1)

Last updated on AUGUST 17, 2023

Applies to:

Purpose

LACP is an IEEE standard for Ethernet (802.1AX-2008). It provides a method for network switches and interfaces to be combined together to increase availability and throughput. It does not increase the overall network speed, as the fastest single transfer will still be limited by the speed of an individual network interface card (NIC), but does increase the throughput by ganging together the individual interfaces so multiple processes can send or receive at the same time. It accomplishes this goal by creating a symbiotic relationship between the ganged network interfaces and the switch(s). Originally LACP was designed to connect multiple switches together into a single entity using standard Ethernet cables. As this technology matured it was extended to host-to-switch connectivity and eventually to host to dual switch configurations. Dual switch configurations require more abilities on the switch side, such has communication between switches, and not all switches will support such a configuration.

Link Aggregation is different from the other forms of bonding (Linux bonding, IPMP, and others) because of the NIC driver's interaction with the network switch. Bonding drivers usually do not require the switch to participate with them, while Link Aggregation does. The NIC’s interaction with the switch occurs via Ethernet packets (Link Aggregation Control Protocol Data Unit or LACPDU) that communicate the aggregation group identification, availability, and other information. The frequency of this packet exchange is determined by the Linux bonding option parameter lacp_rate. This parameter is set to either 0 or 1, which represents the only intervals specified in the IEEE specifications. For 0, an LACPDU packet is sent every thirty seconds while a 1 will have one sent every second. This should have an effect as to when the entire system detects if a link in the aggregation has failed. This interval needs to be set on the switch too. See the switch documentation on how to set this interval.

In Link Aggregation, not only is availability added, but load balancing too. Load balancing is based on a few algorithms. These are designed to limit or eliminate out-of-order frames. If frames are received out of order, TCP will have to spend processing time reordering them. The Linux bonding driver (mode 4) and switch do not negotiate the type of load balancing that will be used between them, so each need to be configured separately. The Linux mode4 bonding is responsible for load balancing the outbound traffic from the node, while the switch is responsible for the inbound load balancing. This would be expected due to how the Address Resolution Protocol (ARP) functions. OS's need to translate Internet Protocol (IP) addresses into Ethernet addresses so the Ethernet packets can be formed and sent. It caches this information in its ARP table. This table will only have one Media Access Control (MAC) address per IP address. An aggregated link, on the other hand, will have multiple NIC's that would have unique MAC addresses. The Linux bonding mode4 driver, when it aggregates the interfaces, gives them all the same MAC address. This results in the one MAC address for an IP address in the OS's ARP table. So when a host receives a packet from an aggregated link, it will always be from a singular MAC regardless of which physical interface it originated from.

It must be pointed out that when LACP is configured on the switch, it must be configured in the bonding driver also. With the switch configured with LACP, no communication was possible until the bonding driver was configured for LACP. LACP is a protocol that both the switch and network interfaces use to gain membership in the aggregation group. Without that membership negotiation, the switch will not forward traffic to the network interface.

Scope

Details