IDT switch on the PCI riser has a problem resulting in occasional loss of connectivity to pair of flash cards on the cells (Doc ID 1351559.1)

Last updated on SEPTEMBER 09, 2016

Applies to:

Oracle Exadata Hardware - Version 11.2.0.1 to 11.2.0.2 [Release 11.2]
Oracle Exadata Storage Server Software - Version 11.2.1.2.0 to 11.2.2.3.5 [Release 11.2]
Information in this document applies to any platform.

Symptoms

Customers will receive an alert titled 'Flash disk removed' (see sample below) when a flash disk has been removed or is no longer visible to Exadata.   

36_1 2011-08-24T16:23:03-07:00 critical "Flash disk removed. Status : NOT PRESENT Manufacturer : MARVELL Model Number : SD88SA02 Size : 23G Serial Number : 0929M00PH9 Firmware : D20Y Slot Number : PCI Slot: 1; FDOM: 0 Cell Disk : FD_00_dscgihcel11 Grid Disk : No grid disks exist Flash Cache : Present Error Count : 0 Last Failure : unknown"
If a flash disk has just been replaced, this alert is informational only and an alert titled 'Flash disk replaced' will be sent shortly. No action is needed.
 
If the flash disk is being used as Exadata Smart Flash Cache then the only effect is that the cache size has been reduced on this cell.  The flash disk removal will not cause an outage, data loss, or ASM rebalance.
 
Each flash card contains four flash disks.  If all four flash disks on a single card produce alert 'Flash disk removed', it is likely due to removal of the flash card.  To see whether the flash disks are on the same flash card, check for identical PCI Slot numbers in the alert outputs.
 
If all the flash disks across two flash cards on the same PCI riser card (PCI slots 2 and 5 share the same riser card; PCI slots 1 and 4 share the same riser card) alert at the same time, then the likely cause is an issue in the PCI riser card that the two flash cards plug into.  In this case it is very likely a transient riser card problem reported in unpublished BUG 15679441.  An example of the alerts reported when this condition is encountered is shown below.  Note that there are eight (8) flash disks reported removed, four (4) on PCI Slot 1 and four (4) on PCI Slot 4.  PCI slots 1 and 4 share the same PCI riser card.

# cellcli -e list alerthistory | grep 'Flash disk removed'
36_1 2011-08-24T16:23:03-07:00 critical "Flash disk removed. Status : NOT PRESENT Manufacturer : MARVELL Model Number : SD88SA02 Size : 23G Serial Number : 0929M00PH9 Firmware : D20Y Slot Number : PCI Slot: 1; FDOM: 0 Cell Disk : FD_00_dscgihcel11 Grid Disk : No grid disks exist Flash Cache : Present Error Count : 0 Last Failure : unknown"

37_1 2011-08-24T16:23:03-07:00 critical "Flash disk removed. Status : NOT PRESENT Manufacturer : MARVELL Model Number : SD88SA02 Size : 23G Serial Number : 0929M00P7C Firmware : D20Y Slot Number : PCI Slot: 1; FDOM: 1 Cell Disk : FD_01_dscgihcel11 Grid Disk : No grid disks exist Flash Cache : Present Error Count : 0 Last Failure : unknown"

38_1 2011-08-24T16:23:03-07:00 critical "Flash disk removed. Status : NOT PRESENT Manufacturer : MARVELL Model Number : SD88SA02 Size : 23G Serial Number : 0929M00PH0 Firmware : D20Y Slot Number : PCI Slot: 1; FDOM: 2 Cell Disk : FD_02_dscgihcel11 Grid Disk : No grid disks exist Flash Cache : Present Error Count : 0 Last Failure : unknown"

39_1 2011-08-24T16:23:03-07:00 critical "Flash disk removed. Status : NOT PRESENT Manufacturer : MARVELL Model Number : SD88SA02 Size : 23G Serial Number : 0929M00PL1 Firmware : D20Y Slot Number : PCI Slot: 1; FDOM: 3 Cell Disk : FD_03_dscgihcel11 Grid Disk : No grid disks exist Flash Cache : Present Error Count : 0 Last Failure : unknown"

40_1 2011-08-24T16:23:03-07:00 critical "Flash disk removed. Status : NOT PRESENT Manufacturer : MARVELL Model Number : SD88SA02 Size : 23G Serial Number : 0924M00N60 Firmware : D20Y Slot Number : PCI Slot: 4; FDOM: 0 Cell Disk : FD_08_dscgihcel11 Grid Disk : No grid disks exist Flash Cache : Present Error Count : 0 Last Failure : unknown"

41_1 2011-08-24T16:23:03-07:00 critical "Flash disk removed. Status : NOT PRESENT Manufacturer : MARVELL Model Number : SD88SA02 Size : 23G Serial Number : 0924M00N64 Firmware : D20Y Slot Number : PCI Slot: 4; FDOM: 1 Cell Disk : FD_09_dscgihcel11 Grid Disk : No grid disks exist Flash Cache : Present Error Count : 0 Last Failure : unknown"

42_1 2011-08-24T16:23:03-07:00 critical "Flash disk removed. Status : NOT PRESENT Manufacturer : MARVELL Model Number : SD88SA02 Size : 23G Serial Number : 0924M00N6G Firmware : D20Y Slot Number : PCI Slot: 4; FDOM: 2 Cell Disk : FD_10_dscgihcel11 Grid Disk : No grid disks exist Flash Cache : Present Error Count : 0 Last Failure : unknown"

43_1 2011-08-24T16:23:03-07:00 critical "Flash disk removed. Status : NOT PRESENT Manufacturer : MARVELL Model Number : SD88SA02 Size : 23G Serial Number : 0924M00N66 Firmware : D20Y Slot Number : PCI Slot: 4; FDOM: 3 Cell Disk : FD_11_dscgihcel11 Grid Disk : No grid disks exist Flash Cache : Present Error Count : 0 Last Failure : unknown"
 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms