Hitachi Data Systems (HDS) SVPmon shows IO Bottleneck Issues Which They Claim is Caused by Server HBA Limitation (Doc ID 2567112.1)

Last updated on OCTOBER 05, 2022

Applies to:

Symptoms

HDS requested the customer to investigate IO bottleneck issue.

HDS wrote:

First of all, there is no hardware issue or performance bottleneck found on internal array which might be related.
Instead, with the SVPmon data in the dump, we could see clearly what is causing the damage..
With SVPmon data in 1-minute interval, the read transfer rate on all the reported ports (5678AB) were sustained at 400MB/s as a flat line as highlighted below.
Usually, such situation is taken as the indication of the bottleneck on the data transfer in read direction, most likely to be related with the HBA port limitation.

In total, it is up to 3.5GB/s in read direction to the reported server. Such high read workload would have chance to trigger the HBA port saturation.
As a result, the read latency was badly impacted on those ports which was perfectly in line with the excessive read workload, especially on port 7A/8A!
The write latency was impacted similarly, highest latency found on ports 7A/8A as well. Both read and write IOs being impacted are also one of the typical symptom once HBA ports reaching its physical limitation in read direction.

What is more, with SVPmon data in 1-hour interval, we could see that similar workload was actually occurring on daily basis over last month
As a result, the high latency issue occurred every day and it is always ports 7A/8A experiencing the highest latency<85> The alignment between the excessive read workload and the higher latency is conclusive.

Server HBA  to HDS port mapping:

Switch fabric 0:

HBA Controller/PWWN      HDS Port          HDS  PWWN

2100000E1EC8C1D0         5A                   50060E8007E3DC40

2100000E1EC8C230         7A                   50060E8007E3DC60

2100000E1EC8C220         5B                   50060E8007E3DC41

2100000E1EC8C0F0         7B                   50060E8007E3DC61

Switch Fabric 1:

HBA Controller PWWN      HDS Port          HDS  PWWN

2100000E1EC8C1D1       6A                   50060E8007E3DC50

2100000E1EC8C231        8A                   50060E8007E3DC70

2100000E1EC8C221        6B                   50060E8007E3DC51

2100000E1EC8C0F1         8B                  50060E8007E3DC71

Changes

Cause

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.