My Oracle Support Banner

ODA Virtualized Platform Performance is Very Slow for X5-2 and X6-2 HA: High Ping Latency on the IB Interconnect ; High GC Waits ; Due to ODA_BASE and VMs Sharing the Same vCPUs (Doc ID 2530968.1)

Last updated on FEBRUARY 20, 2020

Applies to:

Oracle Database Appliance Software - Version 12.1.2 to 18.1 [Release 12.1 to 12.2]
Oracle Database Appliance X5-2 - Version All Versions to All Versions [Release All Releases]
Oracle Database Appliance X6-2 HA Hardware - Version All Versions to All Versions [Release All Releases]
Information in this document applies to any platform.

Symptoms

This note is a copy of the Linux Note on the same subject:

ODA Base Virtual Machine Experiences High Latency On The Private Network (Doc ID 2516151.1)

Minor alterations have been added to this note to add an ODA perspective and to improve find-ability based on associated symptoms

This problem is exclusive to:

  • ODA X5-2 and X6-2
  • Using the INFINIBAND interconnect
  • ODA Virtualized Platform (ODAVP)

 

Some ODA base virtual guest machines experience high latency on the private network which is used for the RAC cluster interconnect. 
The private network ping latency is expected to be less than 1 ms, else RAC cluster nodes can show poor performances. 
The private network is used for intercommunication between RAC nodes and transfer of I/O blocks, hence its latency is expected in < 1 ms.
Note: The interconnect private IPs are set to hard values during the ODA deployment.


 ping(8) stats show 6 to 10 ms or higher latency from one node to other on private network.

Node0


# ifconfig | grep 192.

  inet addr:192.xxx.xxx.28  Bcast:192.xxx.xxx.255 Mask:255.xxx.xxx.0
  RX bytes:978828714 (933.4 MiB) TX bytes:119231104076 (111.0 GiB)

# ping 192.xxx.xxx.27
PING 192.xxx.xxx.27 (192.xxx.xxx.27) 56(84) bytes of data.
64 bytes from 192.xxx.xxx.27: icmp_seq=1 ttl=64 time=2.33 ms
64 bytes from 192.xxx.xxx.27: icmp_seq=2 ttl=64 time=6.78 ms  
64 bytes from 192.xxx.xxx.27: icmp_seq=3 ttl=64 time=0.218 ms
64 bytes from 192.xxx.xxx.27: icmp_seq=4 ttl=64 time=0.188 ms
64 bytes from 192.xxx.xxx.27: icmp_seq=5 ttl=64 time=7.38 ms  
64 bytes from 192.xxx.xxx.27: icmp_seq=6 ttl=64 time=6.34 ms
64 bytes from 192.xxx.xxx.27: icmp_seq=7 ttl=64 time=0.251 ms
64 bytes from 192.xxx.xxx.27: icmp_seq=8 ttl=64 time=4.49 ms  
64 bytes from 192.xxx.xxx.27: icmp_seq=9 ttl=64 time=0.224 ms
64 bytes from 192.xxx.xxx.27: icmp_seq=10 ttl=64 time=1.56 ms
64 bytes from 192.xxx.xxx.27: icmp_seq=11 ttl=64 time=13.8 ms  
64 bytes from 192.xxx.xxx.27: icmp_seq=12 ttl=64 time=2.94 ms
64 bytes from 192.xxx.xxx.27: icmp_seq=13 ttl=64 time=0.203 ms
64 bytes from 192.xxx.xxx.27: icmp_seq=14 ttl=64 time=0.245 ms
64 bytes from 192.xxx.xxx.27: icmp_seq=15 ttl=64 time=0.258 ms
64 bytes from 192.xxx.xxx.27: icmp_seq=16 ttl=64 time=8.47 ms  
64 bytes from 192.xxx.xxx.27: icmp_seq=17 ttl=64 time=0.541 ms
64 bytes from 192.xxx.xxx.27: icmp_seq=18 ttl=64 time=9.28 ms  
node1

# ifconfig | grep 192.

  inet addr:192.xxx.xxx.27 Bcast:192.xxx.xxx.255 Mask:255.xxx.xxx.0


# ping 192.xxx.xxx.28

PING 192.xxx.xxx.28 (192.xxx.xxx.28) 56(84) bytes of data.
64 bytes from 192.xxx.xxx.28: icmp_seq=1 ttl=64 time=0.302 ms
64 bytes from 192.xxx.xxx.28: icmp_seq=2 ttl=64 time=0.211 ms
64 bytes from 192.xxx.xxx.28: icmp_seq=3 ttl=64 time=0.520 ms
64 bytes from 192.xxx.xxx.28: icmp_seq=4 ttl=64 time=16.9 ms   
64 bytes from 192.xxx.xxx.28: icmp_seq=5 ttl=64 time=5.94 ms
64 bytes from 192.xxx.xxx.28: icmp_seq=6 ttl=64 time=0.384 ms
64 bytes from 192.xxx.xxx.28: icmp_seq=7 ttl=64 time=5.23 ms   
64 bytes from 192.xxx.xxx.28: icmp_seq=8 ttl=64 time=0.200 ms
64 bytes from 192.xxx.xxx.28: icmp_seq=9 ttl=64 time=8.05 ms   
64 bytes from 192.xxx.xxx.28: icmp_seq=10 ttl=64 time=0.260 ms
64 bytes from 192.xxx.xxx.28: icmp_seq=11 ttl=64 time=17.2 ms  
64 bytes from 192.xxx.xxx.28: icmp_seq=12 ttl=64 time=0.225 ms
64 bytes from 192.xxx.xxx.28: icmp_seq=13 ttl=64 time=0.316 ms
64 bytes from 192.xxx.xxx.28: icmp_seq=14 ttl=64 time=6.49 ms  
64 bytes from 192.xxx.xxx.28: icmp_seq=15 ttl=64 time=1.47 ms
64 bytes from 192.xxx.xxx.28: icmp_seq=16 ttl=64 time=4.66 ms 

Ping latency can be observed on the dom0 bridge interfaces which are used for the guest private network.


A second symptom is the very high times seen for Dom1 (ODA_BASE) when compared with Dom0 
Proportionately, you will see the Dom-0 has much lower Time(s) than Dom-1 (ODA_BASE)

In the following example we see:

 the Time elapsed for Dom-0 CPU is in the 10k second range     (28,412.1 up to   42,051.5 seconds)
 the Time elapsed for Dom-1 CPU is in the 100k second range  (301,185.3 up to 398,397.9 seconds)

 

Example

 

Changes

 

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


In this Document
Symptoms
Changes
Cause
Solution
References

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.