Running Spark Using Hue Oozie on BDA 4.2 (CDH 5.4.0) Fails with java.lang.ClassNotFoundException Error (Doc ID 2072057.1)

Last updated on NOVEMBER 11, 2015

Applies to:

Big Data Appliance Integrated Software - Version 4.2.0 and later
Linux x86-64

Symptoms

Running spark jobs as "spark shell" or YARN Client mode on BDA works fine, however it does not work when using Hue Oozie.

The Oozie spark job fails with the following error:

Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, Job aborted due to stage failure: Task 1 in stage 0.0 failed 4 times, most recent failure: Lost task 1.3 in stage 0.0 (TID 6,bdanode06.example.com): java.lang.ClassNotFoundException: com.oreilly.learningsparkexamples.mini.java.WordCount$1
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.spark.serializer.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:65)

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms