Bad:The DataNode Has 3 Volume Failure(s) Critical Threshold 2 Volume(s) (Doc ID 2596086.1)

Last updated on APRIL 17, 2023

Applies to:

Symptoms

The Cloudera Manager shows the DataNode with "bad health". The message is: "Bad:The DataNode Has 3 Volume Failure(s) Critical Threshold 2 Volume(s)".

Big Data Appliance Image Info:
IMAGE_LABEL : BDA_4.11.0_LINUX.X64_RELEASE
IMAGE_VERSION : 4.11.0
LINUX_VERSION : Oracle Linux Server release 6.9
KERNEL_VERSION : 4.1.12-103.9.7.el6uek.x86_64
HADOOP_VERSION : 2.6.0-cdh5.13.1

The following tests show that both the hardware and the software are working properly.

++ [root@xx~]# bdachecksw

...

SUCCESS: Big Data Appliance software validation checks succeeded
[root@xx~]#

++[root@xx~]# bdacheckhw

...

SUCCESS: Big Data Appliance hardware validation checks succeeded

[root@xx~]#

++ -bash-4.1$ hdfs fsck /

The filesystem under path '/' is HEALTHY
-bash-4.1$

After checking the DN logs, it seems below 3 disks are marked as failed. However, the 3 volumes/disks seems fine and no errors seen for the volumes.
~~~~~
/dev/sdb4 /u02 ext4 rw,nodev,noatime,data=ordered 0 0
/dev/sdc1 /u03 ext4 rw,nodev,noatime,data=ordered 0 0
/dev/sdd1 /u04 ext4 rw,nodev,noatime,data=ordered 0 0
~~~~~

[..]
"volfails\":3,
"failedStorageLocations\":[\"/u02/hadoop/dfs\",\"/u03/hadoop/dfs\",\"/u04/hadoop/dfs\"], <===
[..]

dfs_data_dir_list
/u12/hadoop/dfs,/u11/hadoop/dfs,/u10/hadoop/dfs,/u09/hadoop/dfs,/u08/hadoop/dfs,/u07/hadoop/dfs,/u06/hadoop/dfs,/u05/hadoop/dfs,/u04/hadoop/dfs,/u03/hadoop/dfs,/u02/hadoop/dfs,/u01/hadoop/dfs

Cause

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.