Executing a Spark Job on BDA V4.5 (Spark-on-Yarn) Fails with "org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow"
(Doc ID 2143437.1)
Last updated on JANUARY 28, 2020
Applies to:
Big Data Appliance Integrated Software - Version 4.5.0 and laterLinux x86-64
Symptoms
Executing a Spark Job on BDA V4.5/CDH 5.7.0 (Spark-on-Yarn) fails with "org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow". The output looks like:
6 WARN scheduler.TaskSetManager: Lost task 0.3 in stage 2.0 (TID <TID>, <HOSTNAME0x>.<DOMAIN>): org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow. Available: 0, required: 3. To avoid this, increase spark.kryoserializer.buffer.max value.
at org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:299)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:240)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
at org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:299)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:240)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Cause
To view full details, sign in with your My Oracle Support account. |
|
Don't have a My Oracle Support account? Click to get started! |
In this Document
Symptoms |
Cause |
Solution |