My Oracle Support Banner

BDA Hive Queries Fail with Exception 'java.io.IOException - Unable to Close File...Last Block Does Not have Enough Number of Replicas" (Doc ID 2438648.1)

Last updated on APRIL 17, 2023

Applies to:

Big Data Appliance Integrated Software - Version 4.11.0 and later
Linux x86-64

Symptoms

NOTE: In the examples that follow, user details, table name, company name, email, hostnames, etc. represent a fictitious sample (and are used to provide an illustrative example only). Any similarity to actual persons, or entities, living or dead, is purely coincidental and not intended in any manner.

When running hive queries (via Oozie, Hue or CLI) on a BDA 4.X clusters, the queries randomly fail with errors like:

Job Submission failed with exception 'java.io.IOException(Unable to close file because the last block BP-395578344-<PRIVATE_IP>-1407857035874:blk_1656069950_584375008 does not have enough number of replicas.)' FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask

...
Aug 3, 2018 9:39:56 AM INFO: parquet.hadoop.InternalParquetRecordReader: block read in memory in 53 ms. row count = 2032
Execution completed successfully
MapredLocal task succeeded
Launching Job 1 out of 10
Number of reduce tasks is set to 0 since there's no reduce operator
java.io.IOException: Unable to close file because the last block BP-395578344-<PRIVATE_IP>-1407857035874:blk_1656069950_584375008 does not have enough number of replicas.
at org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:2760)
at org.apache.hadoop.hdfs.DFSOutputStream.closeImpl(DFSOutputStream.java:2722)
at org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:2681)
at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:63)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:121)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:369)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:341)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:292)
at org.apache.hadoop.mapreduce.JobResourceUploader.copyRemoteFiles(JobResourceUploader.java:203)
at org.apache.hadoop.mapreduce.JobResourceUploader.uploadFiles(JobResourceUploader.java:128)
at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:99)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:194)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1307)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1304)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
...
Job Submission failed with exception 'java.io.IOException(Unable to close file because the last block BP-395578344-<PRIVATE_IP>-1407857035874:blk_1656069950_584375008 does not have enough number of replicas.)'
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask

This is a new problem and did not happen previously. It just starts happening without making any changes on the cluster.

The log files show that the queries complete the execution process but fail in the mapreduce phase. The HDFS snapshot shows the system is healthy and the replication factor is 3.

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!

In this Document
Symptoms
Cause
Solution


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.