My Oracle Support Banner

Spark Job Fails in a Big Data Cloud (BDC) Cluster with "java.io.IOException: java.lang.InterruptedException" (Doc ID 2493419.1)

Last updated on APRIL 02, 2020

Applies to:

Oracle Big Data Cloud Service - Compute Edition - Version N/A and later
Linux x86-64

Symptoms

A Spark job fails in a Big Data Cloud (BDC) cluster as below:

java.io.IOException: java.lang.InterruptedException
  at org.apache.hadoop.ipc.Client.call(Client.java:1411)
  at org.apache.hadoop.ipc.Client.call(Client.java:1363)
  at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
  at com.sun.proxy.$Proxy15.getBlockLocations(Unknown Source)
  at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeP rotocolTranslatorPB.java:257)
  at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:498)
  at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:256)
  at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
  at com.sun.proxy.$Proxy16.getBlockLocations(Unknown Source)
  at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1240)
  at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1227)
  at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1215)
  at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:306)
  at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:272)
  at org.apache.hadoop.hdfs.DFSInputStream.(DFSInputStream.java:264)
  at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1540)
  at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:304)
  at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:300)
  at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
  at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:300)
  at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:767)
  at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.extractMetaInfoFromFooter(ReaderImpl.java:355)
  at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.(ReaderImpl.java:316)
  at org.apache.hadoop.hive.ql.io.orc.OrcFile.createReader(OrcFile.java:237)
  at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getReader(OrcInputFormat.java:1204)
  at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:1113)
  at org.apache.spark.rdd.HadoopRDD$$anon$1.liftedTree1$1(HadoopRDD.scala:252)
  at org.apache.spark.rdd.HadoopRDD$$anon$1.(HadoopRDD.scala:251)
  at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:211)
  at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:102)
  at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
  at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
  at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
  at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
  at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
  at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
  at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
  at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
  at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
  at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
  at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
  at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)



Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


In this Document
Symptoms
Cause
Solution
 To turn Spark's speculative execution off in the Spark Shell
 To turn Spark's speculative execution off for the BDC cluster


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.