Exalogic IB: EoIB Slave 1 Pinned To GW01 Switch Goes Down & Up on VMs (Doc ID 2597214.1)

Last updated on JUNE 07, 2021

Applies to:

Symptoms

This is an Exalogic Virtual rack.

Exalogic Networking Architecture ===> As per design of Exalogic networking, In Exalogic Virtual racks, have Dom0 Compute nodes and Guest VMs on top of Dom0 compute nodes. All the Dom0 compute nodes are cabled into 2 IB Switches. Each Dom0 compute node has 2 IB port - 1 port is connected to gw01 and other port is connected to gw02. All the dom0 compute nodes have IPoIB network bonds, each bond with 2 primarys in active/passive node. One primary is active primary and other one is standby primary. On the Guest VMs running on Dom0 compute nodes have EoIB and IPoIB bonds, each bond with 2 primarys in active/passive node. One primary is active primary and other one is standby primary.

On few Guest VM's see that the EoIB bond primary 1 which is pinned to gw01 switch keeps going down and up very frequently. Other IB bonds primary interfaces 1 pinned to the same gw01 switch do not see any problem. Also the dom0 compute nodes on which  the problematic guest VM  is running does not show and of the IB bond interfaces 1 primarys going down and up. Does not appear there is an issue with the underlying switch connectivity as it is just few VMs EoiB bond primary 1 interfaces which is going down and coming up continuously. Also there is no particular VM which always has the issue. The issue is randomly across all the VM's where it is seen on one VM and goes away when the VM is rebooted

The issue is happening due to swiotlb setting being 256MB low on dom0 compute nodes.  Increased the swiotlb setting to 512MB on the dom0 compute nodes. Also rebooted the IB switches and did SM primary failover of the IB Switches, no change.

See these messages in /var/log/messages showing ::

Cause

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.