Alert - ODA: Extreme Slow IO Performance including: Possible Hang, Poor Disk or Diskgroup Performance with Low IO but High CPU Utilitization (Doc ID 2189437.1)

Last updated on MARCH 05, 2023

Applies to:

Symptoms

This problem can have one or more of the following symptoms:

1. The whole system hangs.
2. The RDBMS on the system hangs.
3. All VMs hang.
4. Performance is very slow.
5. ASM Disks are offlined first then may dropped.
6. ASM diskgroup can be dropped.
7. Previous queries or applications can suddenly take 10x usual durations or longer.
8. IO metrics can suddenly spike hitting fractional seconds per IO vs. previous millisecond IO and query times.
9. The problem may appear intermittent. Sometimes this issue can go away after couple hours or days but come back later.

All of these symptoms can be due to one source:

One of more of the disks has poor IO performance.
This is often due to a failing but not completely failed disk.
One key symptom is that the utilization will show as ~100%  but the write/read IO on the disk is very small for example:

From IO Stats you can see one disk hitting 100% utilization but no very little or no IO.

If you see the disk hitting ~100% utilization and little or no IO and you are on ODA 12.1.2.10 or earlier you are likely encountering this problem.

Changes

No specific change required.

Cause