ODA Virtualized Platform Performance is Very Slow for X5-2 and X6-2 HA: High Ping Latency on the IB Interconnect ; High GC Waits ; Due to ODA_BASE and VMs Sharing the Same vCPUs (Doc ID 2530968.1)

Last updated on JULY 20, 2024

Applies to:

Symptoms

Some ODA base virtual guest machines experience high latency on the private network which is used for the RAC cluster interconnect. 
The private network ping latency is expected to be less than 1 ms, else RAC cluster nodes can show poor performances. 
The private network is used for intercommunication between RAC nodes and transfer of I/O blocks, hence its latency is expected in < 1 ms.
Note: The interconnect private IPs are set to hard values during the ODA deployment.


 ping(8) stats show 6 to 10 ms or higher latency from one node to other on private network.

Node0

# ifconfig | grep 192.

  inet addr:192.xxx.xxx.28  Bcast:192.xxx.xxx.255 Mask:255.xxx.xxx.0
  RX bytes:978828714 (933.4 MiB) TX bytes:119231104076 (111.0 GiB)

# ping 192.xxx.xxx.27

PING 192.xxx.xxx.27 (192.xxx.xxx.27) 56(84) bytes of data.
64 bytes from 192.xxx.xxx.27: icmp_seq=1 ttl=64 time=2.33 ms
64 bytes from 192.xxx.xxx.27: icmp_seq=2 ttl=64 time=6.78 ms  
64 bytes from 192.xxx.xxx.27: icmp_seq=3 ttl=64 time=0.218 ms
64 bytes from 192.xxx.xxx.27: icmp_seq=4 ttl=64 time=0.188 ms
64 bytes from 192.xxx.xxx.27: icmp_seq=5 ttl=64 time=7.38 ms  
64 bytes from 192.xxx.xxx.27: icmp_seq=6 ttl=64 time=6.34 ms
64 bytes from 192.xxx.xxx.27: icmp_seq=7 ttl=64 time=0.251 ms
64 bytes from 192.xxx.xxx.27: icmp_seq=8 ttl=64 time=4.49 ms  
64 bytes from 192.xxx.xxx.27: icmp_seq=9 ttl=64 time=0.224 ms
64 bytes from 192.xxx.xxx.27: icmp_seq=10 ttl=64 time=1.56 ms
64 bytes from 192.xxx.xxx.27: icmp_seq=11 ttl=64 time=13.8 ms  
64 bytes from 192.xxx.xxx.27: icmp_seq=12 ttl=64 time=2.94 ms
64 bytes from 192.xxx.xxx.27: icmp_seq=13 ttl=64 time=0.203 ms
64 bytes from 192.xxx.xxx.27: icmp_seq=14 ttl=64 time=0.245 ms
64 bytes from 192.xxx.xxx.27: icmp_seq=15 ttl=64 time=0.258 ms
64 bytes from 192.xxx.xxx.27: icmp_seq=16 ttl=64 time=8.47 ms  
64 bytes from 192.xxx.xxx.27: icmp_seq=17 ttl=64 time=0.541 ms
64 bytes from 192.xxx.xxx.27: icmp_seq=18 ttl=64 time=9.28 ms  
node1

# ifconfig | grep 192.

  inet addr:192.xxx.xxx.27 Bcast:192.xxx.xxx.255 Mask:255.xxx.xxx.0


# ping 192.xxx.xxx.28

PING 192.xxx.xxx.28 (192.xxx.xxx.28) 56(84) bytes of data.
64 bytes from 192.xxx.xxx.28: icmp_seq=1 ttl=64 time=0.302 ms
64 bytes from 192.xxx.xxx.28: icmp_seq=2 ttl=64 time=0.211 ms
64 bytes from 192.xxx.xxx.28: icmp_seq=3 ttl=64 time=0.520 ms
64 bytes from 192.xxx.xxx.28: icmp_seq=4 ttl=64 time=16.9 ms   
64 bytes from 192.xxx.xxx.28: icmp_seq=5 ttl=64 time=5.94 ms
64 bytes from 192.xxx.xxx.28: icmp_seq=6 ttl=64 time=0.384 ms
64 bytes from 192.xxx.xxx.28: icmp_seq=7 ttl=64 time=5.23 ms   
64 bytes from 192.xxx.xxx.28: icmp_seq=8 ttl=64 time=0.200 ms
64 bytes from 192.xxx.xxx.28: icmp_seq=9 ttl=64 time=8.05 ms   
64 bytes from 192.xxx.xxx.28: icmp_seq=10 ttl=64 time=0.260 ms
64 bytes from 192.xxx.xxx.28: icmp_seq=11 ttl=64 time=17.2 ms  
64 bytes from 192.xxx.xxx.28: icmp_seq=12 ttl=64 time=0.225 ms
64 bytes from 192.xxx.xxx.28: icmp_seq=13 ttl=64 time=0.316 ms
64 bytes from 192.xxx.xxx.28: icmp_seq=14 ttl=64 time=6.49 ms  
64 bytes from 192.xxx.xxx.28: icmp_seq=15 ttl=64 time=1.47 ms
64 bytes from 192.xxx.xxx.28: icmp_seq=16 ttl=64 time=4.66 ms 

Ping latency can be observed on the dom0 bridge interfaces which are used for the guest private network.


A second symptom is the very high times seen for Dom1 (ODA_BASE) when compared with Dom0 
Proportionately, you will see the Dom-0 has much lower Time(s) than Dom-1 (ODA_BASE)

In the following example we see:

 the Time elapsed for Dom-0 CPU is in the 10k second range     (28,412.1 up to   42,051.5 seconds)
 the Time elapsed for Dom-1 CPU is in the 100k second range  (301,185.3 up to 398,397.9 seconds)

Example

Changes

Cause

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.