ODA: SSD shows "unknown" "NewDiskInserted" after HDD replacement (Doc ID 2255282.1)

Last updated on OCTOBER 18, 2024

Applies to:

Oracle Database Appliance Software - Version 12.1.2.6 to 12.1.2.9 [Release 12.1]
Oracle Database Appliance X5-2 - Version All Versions and later
Oracle Database Appliance X4-2 - Version All Versions and later
Oracle Database Appliance X3-2 - Version All Versions and later
Oracle Database Appliance - Version All Versions and later
Linux x86-64

Symptoms

NOTE: In the images, examples and document that follow, user details, cluster names, hostnames, directory paths, filenames, etc. represent a fictitious sample (and are used to provide an illustrative example only). Any similarity to actual persons, or entities, living or dead, is purely coincidental and not intended in any manner.

ODA X5-2 12.1.2.6.0. After customer replaced one of HDD in another slot, SSD state becomes "unknown" "NewDiskInserted"

This note contains aged ODA configuration information.
The following steps may not apply to SSDs / NVMe disks on some newer hardware.
or DCS stack configurations (Bare Metal on 18.x and higher)
- Consult the relevant ODA version documentation specifically for X6-2, X7-2, X8-2 and higher hardware types which can require disk power down / up first.

From oakd.log , the following message returned:

2017-04-04 08:44:05.510: [ ][696231680]{0:21:2} [resource_initialize] INFO: Calling function flushMultipathDiskPath
2017-04-04 08:44:05.510: [ ][696231680]{0:21:2} [resource_initialize] INFO: Running /sbin/multipath -f SSD_E1_S23_1xxxxxxxx4
2017-04-04 08:44:05.510: [ ][696231680]{0:21:2} [resource_initialize] Apr 04 08:44:05 | SSD_E1_S23_1xxxxxxxx4: map in use
2017-04-04 08:44:05.510: [ ][696231680]{0:21:2} [resource_initialize] ERROR: Failure in running /sbin/multipath -f SSD_E1_S23_1xxxxxxxx4, system command return status 256
2017-04-04 08:44:05.510: [ ][696231680]{0:21:2} [resource_initialize] INFO: Calling function dump_lsof_output
2017-04-04 08:44:05.510: [ ][696231680]{0:21:2} [resource_initialize] INFO: running lsof | egrep "SSD_E1_S23_1xxxxxxxx4|/dev/sdaf|/dev/sdt"
2017-04-04 08:44:06.161: [ ][696231680]{0:21:2} [resource_initialize] oakd 3217 root 54r BLK 65,48 0t0 7399 /dev/sdt
2017-04-04 08:44:06.161: [ ][696231680]{0:21:2} [resource_initialize] multipath 3231 root 45r BLK 65,48 0t0 7399 /dev/sdt
2017-04-04 08:44:06.161: [ ][696231680]{0:21:2} [resource_initialize] multipath 3231 root 56r BLK 65,240 0t0 6690 /dev/sdaf
2017-04-04 08:44:06.161: [ ][696231680]{0:21:2} [resource_initialize] oakcli.bi 18593 root 23r BLK 65,48 0t0 7399 /dev/sdt
2017-04-04 08:44:06.161: [ ][696231680]{0:21:2} [resource_initialize] oakcli.bi 18593 root 24r BLK 65,48 0t0 7399 /dev/sdt
2017-04-04 08:44:06.161: [ ][696231680]{0:21:2} [resource_initialize] INFO: Calling function removeDevMapPath
2017-04-04 08:44:06.162: [ ][696231680]{0:21:2} [resource_initialize] INFO: Running /sbin/dmsetup remove -f SSD_E1_S23_1xxxxxxxx4 <<<<<<<<<<<<<<<<<<<<<<<<
2017-04-04 08:44:06.212: [ ][696231680]{0:21:2} [resource_initialize] device-mapper: remove ioctl on SSD_E1_S23_1xxxxxxxx4 failed: Device or resource busy <<<<<<<<<<<<<<<<<<<<<<<<
2017-04-04 08:44:06.212: [ ][696231680]{0:21:2} [resource_initialize] Command failed
2017-04-04 08:44:06.212: [ ][696231680]{0:21:2} [resource_initialize] ERROR: Failure in running /sbin/dmsetup remove -f SSD_E1_S23_1xxxxxxxx4, system command return status 256

OS message file also reports I/O Multipath errors

Apr 4 08:44:02 oda1 oakcli.bin[18593]: OAKCLI : 'oakcli reinit is issued
Apr 4 08:44:02 oda1 oakcli.bin[18593]: OAKCLI: 'oakcli reinit pd_23' is issued
Apr 4 08:44:06 oda1 kernel: end_request: I/O error, dev dm-51, sector 25694926
Apr 4 08:44:07 oda1 kernel: device-mapper: table: 252:72: multipath: error getting device

Followed the document 1461219.1 but it did not help.

Reseat ( pulling or removing a problem disk ) and replace disk did not help.