H2H Fails in "Spark" Execution Mode With 'Error creating session for session kind spark' (Doc ID 2810487.1)

Last updated on MAY 08, 2023

Applies to:

Symptoms

In an OFSAA environment using Hive with Spark, you get the error below when running an H2H data load:

[28-09-20 07:55:42,112 GMT AM] [INFO ] [BACKEND] [<USER>] [DMT] [Postman]~[postTask]~target : HDFS and source : HDFS
[28-09-20 07:55:42,113 GMT AM] [INFO ] [BACKEND] [<USER>] [DMT] [Postman]~[getDefinitionVersion]~Fetching version for T2T_FCT_RECOVERYEXT~HiveStg<INFODOM>DMT_DMT2T
.
.
.
[28-09-20 07:57:52,699 GMT AM] [INFO ] [BACKEND] [NA] [NA] REST Service Invoked Successfully..{"id":0,"appId":null,"owner":null,"proxyUser":"<USER>","state":"shutting_down","kind":"spark","appInfo":{"driverLogUrl":null,"sparkUiUrl":null},"log":["stdout: ","\nstderr: ","WARNING: User-defined SPARK_HOME (/opt/cloudera/parcels/SPARK2-2.4.0.cloudera2-1.cdh5.13.3.p0.1041012/lib/spark2) overrides detected (/opt/cloudera/parcels/SPARK2/lib/spark2).","WARNING: Running spark-class from user-defined location.","\nYARN Diagnostics: "]}
[28-09-20 07:57:52,702 GMT AM] [INFO ] [BACKEND] [NA] [NA] Session 0 is in state shutting_down , appId null
[28-09-20 07:57:52,702 GMT AM] [ERROR] [BACKEND] [NA] [NA] Error creating session for session kind spark
java.lang.Exception: Session 0 creation timedout in 120 seconds, appId: null, log: [stdout: ,
stderr: , WARNING: User-defined SPARK_HOME (/opt/cloudera/parcels/SPARK2-2.4.0.cloudera2-1.cdh5.13.3.p0.1041012/lib/spark2) overrides detected (/opt/cloudera/parcels/SPARK2/lib/spark2)., WARNING: Running spark-class from user-defined location.,
YARN Diagnostics: ]
     at com.ofs.aai.gateway.client.OfsaaLivyClient.createSession(OfsaaLivyClient.java:427) [aai-data-gateway.jar:?]
     at com.ofs.aai.gateway.client.OfsaaLivyClient.getOrCreateSession(OfsaaLivyClient.java:374) [aai-data-gateway.jar:?]
     at com.ofs.aai.gateway.client.OfsaaLivyClient.getOrCreateSession(OfsaaLivyClient.java:317) [aai-data-gateway.jar:?]
     at com.ofs.aai.gateway.client.OfsaaLivyClient.getOrCreateSession(OfsaaLivyClient.java:300) [aai-data-gateway.jar:?]
     at com.ofs.aai.engine.dmt.dao.impl.load.spark.T2TSparkExecutorDAO.executePreLoading(T2TSparkExecutorDAO.java:428) [aai-dmt-engine.jar:?]
     at com.ofs.aai.engine.dmt.TaskManager.execute(TaskManager.java:138) [aai-dmt-engine.jar:?]
     at com.ofs.aai.engine.dmt.Postman.invokeTask(Postman.java:536) [aai-dmt-engine.jar:?]
     at com.ofs.aai.engine.dmt.Postman.postTask(Postman.java:453) [aai-dmt-engine.jar:?]
     at com.ofs.aai.engine.dmt.Postman.main(Postman.java:115) [aai-dmt-engine.jar:?]

The error prevents you from running H2H loads.

Cause

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.