My Oracle Support Banner

Bad:The DataNode Has 3 Volume Failure(s) Critical Threshold 2 Volume(s) (Doc ID 2596086.1)

Last updated on JANUARY 06, 2022

Applies to:

Big Data Appliance Integrated Software - Version 4.11.0 and later
Information in this document applies to any platform.

Symptoms

The Cloudera Manager shows the DataNode with "bad health". The message is: "Bad:The DataNode Has 3 Volume Failure(s) Critical Threshold 2 Volume(s)".

 

Big Data Appliance Image Info:
IMAGE_LABEL : BDA_4.11.0_LINUX.X64_RELEASE
IMAGE_VERSION : 4.11.0
LINUX_VERSION : Oracle Linux Server release 6.9
KERNEL_VERSION : 4.1.12-103.9.7.el6uek.x86_64
HADOOP_VERSION : 2.6.0-cdh5.13.1

 

The following tests show that both the hardware and the software are working properly.

++ [root@xx~]# bdachecksw

...

SUCCESS: Big Data Appliance software validation checks succeeded
[root@xx~]#

++[root@xx~]# bdacheckhw

...

SUCCESS: Big Data Appliance hardware validation checks succeeded

[root@xx~]#

 

++ -bash-4.1$ hdfs fsck /

The filesystem under path '/' is HEALTHY
-bash-4.1$

 

After checking the DN logs, it seems below 3 disks are marked as failed. However, the 3 volumes/disks seems fine and no errors seen for the volumes.
~~~~~
/dev/sdb4 /u02 ext4 rw,nodev,noatime,data=ordered 0 0
/dev/sdc1 /u03 ext4 rw,nodev,noatime,data=ordered 0 0
/dev/sdd1 /u04 ext4 rw,nodev,noatime,data=ordered 0 0
~~~~~

[..]
"volfails\":3,
"failedStorageLocations\":[\"/u02/hadoop/dfs\",\"/u03/hadoop/dfs\",\"/u04/hadoop/dfs\"], <===
[..]

dfs_data_dir_list
/u12/hadoop/dfs,/u11/hadoop/dfs,/u10/hadoop/dfs,/u09/hadoop/dfs,/u08/hadoop/dfs,/u07/hadoop/dfs,/u06/hadoop/dfs,/u05/hadoop/dfs,/u04/hadoop/dfs,/u03/hadoop/dfs,/u02/hadoop/dfs,/u01/hadoop/dfs

 

 

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


In this Document
Symptoms
Cause
Solution
References


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.