Understanding the HADOOP_MAPRED_HOME Environment Variable on BDA V4 (Doc ID 2019741.1)

Last updated on JUNE 12, 2015

Applies to:

Big Data Appliance Integrated Software - Version 4.1.0 and later
Linux x86-64

Goal

1. The Cloudera documentation on "Setting HADOOP_MAPRED_HOME" explains that:

For MapReduce v2 (YARN):

HADOOP_MAPRED_HOME=/usr/lib/hadoop-mapreduce


For MapReduce v1 (MRv1):

HADOOP_MAPRED_HOME=/usr/lib/hadoop-0.20-mapreduce



2. On BDA V4.1 and v4.2, where MapReduce 2.0 (MRv2) running on YARN is the only supported option,  HADOOP_MAPRED_HOME is the opposite:

HADOOP_MAPRED_HOME=/opt/cloudera/parcels/CDH/lib/hadoop-0.20-mapreduce



In older BDA releases supported MR1 HADOOP_MAPRED_HOME was:

HADOOP_MAPRED_HOME=/usr/lib/hadoop-0.20-mapreduce



3. Testing shows that submitting a job using "yarn jar" vs "hadoop jar" seems  to work with either setting of the HADOOP_MAPRED_HOME environment variable.

For example these scenarios are successful on BDA V4.*

export HADOOP_MAPRED_HOME=/opt/cloudera/parcels/CDH/lib/hadoop-0.20-mapreduce
yarn jar $HADOOP_MAPRED_HOME/hadoop-examples-*-mr1-*.jar pi 10 100
hadoop jar $HADOOP_MAPRED_HOME/hadoop-examples-*-mr1-*.jar pi 10 100

 

export HADOOP_MAPRED_HOME=/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce
yarn jar $HADOOP_MAPRED_HOME/hadoop-mapreduce-examples-*.jar pi 10 100
hadoop jar $HADOOP_MAPRED_HOME/hadoop-examples-*-mr1-*.jar pi 10 100



So the question is what is the reasoning behind HADOOP_MAPRED_HOME on BDA V4.*.  

Solution

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms