Data Node in Bad Health on Oracle Big Data Appliance with "java.io.IOException: Premature EOF from inputStream" (Doc ID 2149174.1)

Last updated on JULY 17, 2017

Applies to:

Big Data Appliance Integrated Software - Version 4.3.0 and later
Linux x86-64

Symptoms

A few times a day, a DataNode gets into "bad" health and the data on it can not be used.  The charts show that the data on that DataNode is under replicated and the DataNode has less memory than expected.  The error resolves itself after a few minutes.  The system has a lot of small files which might contribute the problem.

The following is seen in the log file:

--
2016-06-03 07:22:01,852 INFO org.apache.hadoop.util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1758ms GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=5762ms
2016-06-03 07:22:01,851 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.10.100:47492, dest: /192.168.10.108:1004, bytes: 87027, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-999679297_57, offset: 0, srvID: 66c646ef-c9fd-4a24-81d0-8d37cfc17fd2, blockid: BP-117273040-192.168.10.100-1444230114544:blk_1081013722_7331619, duration: 5776024333
2016-06-03 07:22:01,851 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.10.109:38920, dest: /192.168.10.108:1004, bytes: 18111, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-580985732_60, offset: 0, srvID: 66c646ef-c9fd-4a24-81d0-8d37cfc17fd2, blockid: BP-117273040-192.168.10.100-1444230114544:blk_1081013714_7331611, duration: 5877739743
--
2016-06-03 07:22:13,693 INFO org.apache.hadoop.util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 5544ms GC pool 'ConcurrentMarkSweep' had collection(s): count=4 time=11359ms
2016-06-03 07:22:13,692 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Exception for BP-117273040-192.168.10.100-1444230114544:blk_1081013728_7331625
java.io.IOException: Premature EOF from inputStream
--
..

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms