A Single Disk Replacement can cause Performance Issues on entire Exadata System (Doc ID 3059053.1)

Last updated on MARCH 06, 2025

Applies to:

Symptoms

A Single disk ( Hard Disk Or Flash Disk )  replacement can impact the entire Exadata system and all the databases to perform poorly.

The evidence is seen in AWR under:

For Flash Disk

=====================

• 'FlashCache Space Usage': where the cell that had disk replacement has more %write-only (celadm04 in the example below)
• 'FlashCache Replacements': High OLTP replacement due to the space pressure created by the high write only allocation

The additional IOs cached on the cell create space pressure in the Flashcache, resulting OLTP/Smart scan allocation failures.

For Hard Disk

=====================

1) Replaced drive is physically healthy (Unless it's dead on arrival )

2) The rebalance is also completed ( ASM gv$asm_operation shows NO rows selected)

AND

2) Flash Warm Up is also completed (Exadata  Cell Alert log shows following)

Example :
==========
updateHealth: For GridDisk DATAC1_CD_03_<CELL_NAME> target health GOOD, current health BAD. Reason clearing warmup Inc 2

But you will then start seeing the newly replaced disk becoming a bottleneck

Example:

++ Cell in question and evidence of ongoing performance issue:

CD_IO_TM_W_LG_RQ CD_05_<CELL_NAME> 11,485 us/request
CD_IO_TM_R_LG_RQ CD_05_<CELL_NAME> 117,786 us/request
CD_IO_TM_R_SM_RQ CD_05_<CELL_NAME> 496,116 us/request
CD_IO_LOAD CD_05_<CELL_NAME> 26.0
CD_IO_UTIL CD_05_<CELL_NAME> 100.0 % >>>>>>>

OR

You can see :

CellCLI> list metriccurrent where objectType = 'CELLDISK' and name = 'CD_IO_UTIL'
CD_IO_UTIL CD_00_<CELL_NAME> 14.8 %
CD_IO_UTIL CD_01_<CELL_NAME> 16.0 %
CD_IO_UTIL CD_02_<CELL_NAME> 16.8 %
CD_IO_UTIL CD_03_<CELL_NAME> 14.2 %
CD_IO_UTIL CD_04_<CELL_NAME> 13.5 %
CD_IO_UTIL CD_05_<CELL_NAME> 100.0 % >>>>
CD_IO_UTIL CD_06_<CELL_NAME> 13.9 %
CD_IO_UTIL CD_07_<CELL_NAME> 13.4 %
CD_IO_UTIL CD_08_<CELL_NAME> 12.6 %
CD_IO_UTIL CD_09_<CELL_NAME> 16.3 %
CD_IO_UTIL CD_10_<CELL_NAME> 15.9 %
CD_IO_UTIL CD_11_<CELL_NAME> 14.3 %
CD_IO_UTIL FD_00_<CELL_NAME> 5.9 %
CD_IO_UTIL FD_01_<CELL_NAME> 6.6 %
CD_IO_UTIL FD_02_<CELL_NAME> 6.8 %
CD_IO_UTIL FD_03_<CELL_NAME> 6.5 %

OR

 AND

Disk Utilization is high

Changes

 Replaced disk. 

Cause

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.