Running a spark-shell Command From a Edge Node Fails with java.io.IOException: Cannot run program "/etc/hadoop/conf.cloudera.yarn/topology.py" on BDA V4.3 and Higher (Doc ID 2111307.1)

Last updated on FEBRUARY 29, 2016

Applies to:

Big Data Appliance Integrated Software - Version 4.3.0 and later
Linux x86-64

Symptoms

Running a spark-shell command from a client/edge node to a BDA V4.3 or higher BDA cluster fails with:

5/12/18 19:01:51 WARN net.ScriptBasedMapping: Exception running /etc/hadoop/conf.cloudera.yarn/topology.py 10.132.7.213
java.io.IOException: Cannot run program "/etc/hadoop/conf.cloudera.yarn/topology.py" (in directory "/root"): error=2, No such file or directory
  at java.lang.ProcessBuilder.start(ProcessBuilder.java:1041)
  at org.apache.hadoop.util.Shell.runCommand(Shell.java:448)
  at org.apache.hadoop.util.Shell.run(Shell.java:418)
  at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
  at org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.runResolveCommand(ScriptBasedMapping.java:241)
  at org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.resolve(ScriptBasedMapping.java:179)
  at org.apache.hadoop.net.CachedDNSToSwitchMapping.resolve(CachedDNSToSwitchMapping.java:119)
...
Caused by: java.io.IOException: error=2, No such file or directory
  at java.lang.UNIXProcess.forkAndExec(Native Method)
  at java.lang.UNIXProcess.(UNIXProcess.java:135)
  at java.lang.ProcessImpl.start(ProcessImpl.java:130)
  at java.lang.ProcessBuilder.start(ProcessBuilder.java:1022)
  ...




Changes

This happens after the BDA was upgraded to a higher version of CDH than what is on the client/edge node.

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms