Cloudera Manager Reports HDFS in Bad Health on the BDA due to Missing/Corrupt Blocks Related to Oozie SharedLib
(Doc ID 2073967.1)
Last updated on DECEMBER 12, 2019
Applies to:Big Data Appliance Integrated Software - Version 4.2.0 and later
Cloudera Manager (CM) reports HDFS in bad health on the BDA due to missing/corrupt blocks related to the Oozie ShareLib.
1. After following: During Installation DataNode Fails to Start with 'Block pool ID needed' and 'HDFS Partions already locked' Errors on a Cluster with On-Disk Encryption Enabled (Doc ID 1671417.1), on a new installation, "/usr/bin/hadoop fsck /" reports a corrupt filesystem under '/'.
For example: su hdfs -c "/usr/bin/hadoop fsck /", reports:
Total dirs: 2433
Total files: 527
Total symlinks: 0
Total blocks (validated): 526 (avg. block size 1368931 B)
CORRUPT FILES: 526
MISSING BLOCKS: 526
MISSING SIZE: 720057883 B
CORRUPT BLOCKS: 526
Minimally replicated blocks: 0 (0.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 0 (0.0 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 3
Average block replication: 0.0
Corrupt blocks: 526
Missing replicas: 0
Number of data-nodes: 6
Number of racks: 1
FSCK ended at Thu Oct 22 13:20:35 CEST 2015 in 65 milliseconds
The filesystem under path '/' is CORRUPT
2. The MISSING/CORRUPT blocks are all related to the Oozie Sharelib like:
/user/oozie/share/lib/lib_20150701123505/hcatalog/ST4-4.0.4.jar: CORRUPT blockpool BP-*-<IB IP>-* block blk_<X>
/user/oozie/share/lib/lib_20150701123505/sqoop/hive-shims.jar: MISSING 1 blocks of total size 9135 B..
/user/oozie/share/lib/lib_20150701123505/sqoop/hsqldb-188.8.131.52.jar: CORRUPT blockpool BP-*-<IB IP>-* block blk_<X>
/user/oozie/share/lib/lib_20150701123505/sqoop/hsqldb-184.108.40.206.jar: MISSING 1 blocks of total size 706710 B..
/user/oozie/share/lib/lib_20150701123505/sqoop/htrace-core-3.1.0-incubating.jar: CORRUPT blockpool BP-*-<IB IP>-* block blk_<X>
3. From the MOS document, HDFS is in Bad Health Due to Missing/Corrupt Blocks on BDA V4.1 (Doc ID 2042096.1), we know that if the blocks are determined to be temporary they can be removed with:
4. However in this case "hdfs fsck / -files -blocks -locations" shows the blocks are not temporary but are all related to the Oozie Sharelib.
To view full details, sign in with your My Oracle Support account.
Don't have a My Oracle Support account? Click to get started!
In this Document