Cloudera Manager Reports HDFS in Bad Health on the BDA due to Missing/Corrupt Blocks Related to Oozie SharedLib

(Doc ID 2073967.1)

Last updated on SEPTEMBER 28, 2017

Applies to:

Big Data Appliance Integrated Software - Version 4.2.0 and later
Linux x86-64

Symptoms

Cloudera Manager (CM) reports HDFS in bad health on the BDA due to missing/corrupt blocks related to the Oozie ShareLib.

1. After following: During Installation DataNode Fails to Start with 'Block pool ID needed' and 'HDFS Partions already locked' Errors on a Cluster with On-Disk Encryption Enabled (Doc ID 1671417.1), on a new installation, "/usr/bin/hadoop fsck /" reports a corrupt filesystem under '/'.

For example: su hdfs -c "/usr/bin/hadoop fsck /", reports:

Total size: 720057883 B
Total dirs: 2433
Total files: 527
Total symlinks: 0
Total blocks (validated): 526 (avg. block size 1368931 B)
********************************
CORRUPT FILES: 526
MISSING BLOCKS: 526
MISSING SIZE: 720057883 B
CORRUPT BLOCKS: 526
********************************
Minimally replicated blocks: 0 (0.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 0 (0.0 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 3
Average block replication: 0.0
Corrupt blocks: 526
Missing replicas: 0
Number of data-nodes: 6
Number of racks: 1
FSCK ended at Thu Oct 22 13:20:35 CEST 2015 in 65 milliseconds

The filesystem under path '/' is CORRUPT


2. The MISSING/CORRUPT blocks are all related to the Oozie Sharelib like:

/user/oozie/share/lib/lib_20150701123505/distcp/oozie-sharelib-distcp-4.1.0-cdh5.4.0.jar: MISSING 1 blocks of total size 10422 B..
/user/oozie/share/lib/lib_20150701123505/hcatalog/ST4-4.0.4.jar: CORRUPT blockpool BP-*-<IB IP>-* block blk_<X>
...
/user/oozie/share/lib/lib_20150701123505/sqoop/hive-shims.jar: MISSING 1 blocks of total size 9135 B..
/user/oozie/share/lib/lib_20150701123505/sqoop/hsqldb-1.8.0.10.jar: CORRUPT blockpool BP-*-<IB IP>-* block blk_<X>

/user/oozie/share/lib/lib_20150701123505/sqoop/hsqldb-1.8.0.10.jar: MISSING 1 blocks of total size 706710 B..
/user/oozie/share/lib/lib_20150701123505/sqoop/htrace-core-3.1.0-incubating.jar: CORRUPT blockpool BP-*-<IB IP>-* block blk_<X>


3. From the MOS document, HDFS is in Bad Health Due to Missing/Corrupt Blocks on BDA V4.1 (Doc ID 2042096.1), we know that if the blocks are determined to be temporary they can be removed with:


4. However in this case "hdfs fsck / -files -blocks -locations" shows the blocks are not temporary but are all related to the Oozie Sharelib.

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms