Alert - ODA: Extreme Slow IO Performance including: Possible Hang, Poor Disk or Diskgroup Performance with Low CPU and High CPU Utilitization

(Doc ID 2189437.1)

Last updated on MARCH 20, 2018

Applies to:

Oracle Database Appliance Software - Version 2.7.0.0 to 12.1.2.10 [Release 2.7 to 12.1]
Information in this document applies to any platform.
We have seen ODA HW type: V1 X3-2 X4-2 X5-2 and all versions from 2.7 up to 12.1.2.10 can hit this issue.

Symptoms

This problem can have different final symptoms:

1. The whole system hang.
2. The RDBMS on the system hang.
3. All VM hang.
4. Everything is very slow.
5. ASM Disk got offline first then may dropped.
6. ASM diskgroup can be dropped.
7. Previous queries or applications can suddenly take 10x or more longer
8. IO metrics can suddenly spike hitting fractional seconds vs. previous millisecond IO and query times.
9. Sometime the issue can go away after couple hours or days.  And come back after awhile.

But all of these case has one symptom:

One of more of the disks is using 100% util but the write/read IO on the disk is very small for example:

zzz ***Fri Jul 3 07:35:13 BST 2015
sdq 0.00 0.00 0.00 0.00 0.00 0.00 0.00  24.00 0.00 0.00 100.00

Here the 100% is the disk util and there is no io on the system.  It keeps like this for more than 5 minutes mostly they hit the issue. 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms