My Oracle Support Banner

Executing a Spark Job on BDA V4.5 (Spark-on-Yarn) Fails with "org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow" (Doc ID 2143437.1)

Last updated on JANUARY 28, 2020

Applies to:

Big Data Appliance Integrated Software - Version 4.5.0 and later
Linux x86-64

Symptoms

Executing a Spark Job on BDA V4.5/CDH 5.7.0 (Spark-on-Yarn) fails with "org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow".  The output looks like:

6 WARN scheduler.TaskSetManager: Lost task 0.3 in stage 2.0 (TID <TID>, <HOSTNAME0x>.<DOMAIN>): org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow. Available: 0, required: 3. To avoid this, increase spark.kryoserializer.buffer.max value.
  at org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:299)
  at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:240)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  at java.lang.Thread.run(Thread.java:745)

 

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!

In this Document
Symptoms
Cause
Solution


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.