After Adding New Nodes Yarn Applications Raise: ERROR storage.ShuffleBlockFetcherIterator

(Doc ID 2112141.1)

Last updated on MARCH 01, 2016

Applies to:

Big Data Appliance Integrated Software - Version 4.3.0 and later
Linux x86-64

Symptoms

After 12 new nodes are added to the Oracle Big Data Appliance (BDA) cluster yarn applications start raising the errors like below:

16/02/25 13:44:49 ERROR storage.ShuffleBlockFetcherIterator: Failed to get block(s) from <new node>.example.com:30587
16/02/25 13:44:51 ERROR storage.ShuffleBlockFetcherIterator: Failed to get block(s) from <new node>.example.com:27706
16/02/25 13:44:51 ERROR storage.ShuffleBlockFetcherIterator: Failed to get block(s) from <new node>.example.com:22170
16/02/25 13:44:51 ERROR storage.ShuffleBlockFetcherIterator: Failed to get block(s) from <new node>.example.com:22170
16/02/25 13:44:51 ERROR storage.ShuffleBlockFetcherIterator: Failed to get block(s) from <new node>.example.com:22170
.....


Additionally, running Hive jobs raise:

Error for map tasks:
Error: org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z


Reduce tasks raise:

java.lang.RuntimeException at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:392)
at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:295)
at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:181)
at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:224)

 

It is observed that the parameter mapreduce.job.ubertask.enable was changed / enabled during the addition of the new nodes.

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms