My Oracle Support Banner

Executing a Spark Job on BDA V4.5 (Spark-on-Yarn) Fails with "org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow" (Doc ID 2143437.1)

Last updated on JANUARY 28, 2020

Applies to:

Big Data Appliance Integrated Software - Version 4.5.0 and later
Linux x86-64


Executing a Spark Job on BDA V4.5/CDH 5.7.0 (Spark-on-Yarn) fails with "org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow".  The output looks like:

6 WARN scheduler.TaskSetManager: Lost task 0.3 in stage 2.0 (TID <TID>, <HOSTNAME0x>.<DOMAIN>): org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow. Available: 0, required: 3. To avoid this, increase spark.kryoserializer.buffer.max value.
  at org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:299)
  at org.apache.spark.executor.Executor$
  at java.util.concurrent.ThreadPoolExecutor.runWorker(
  at java.util.concurrent.ThreadPoolExecutor$



To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!

In this Document

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.