My Oracle Support Banner

Inferschema Option in LKM HDFS to Spark.GLOBAL does not Infer Schema even when Configured to use Dataframes (Doc ID 2379854.1)

Last updated on FEBRUARY 03, 2019

Applies to:

Oracle Data Integrator - Version 12.2.1.3.0 and later
Information in this document applies to any platform.

Symptoms

In Oracle Data Integrator (ODI) 12.2.1.3, when attempting to use Inferschema with LKM HDFS to Spark.GLOBAL, the schema is not Inferred - even when configured to use Dataframes.

The following error occurs in this situation:

ODI-1590: The execution of the script failed.
ODIKM-SPARK-SYNC-10000: EKM Command Failed with Exception: java.lang.Exception: Traceback (most recent call last):
10 File "/tmp/Spark_Infer_Hdfs_Hive_Test_Physical.py", line 58, in
11 CONTINENTS_INFERRED = CONTINENTS_INFERRED.toDF()
12 File "/opt/cloudera/parcels/SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354/lib/spark2/python/lib/pyspark.zip/pyspark/sql/dataframe.py", line 1685, in toDF
13 File "/opt/cloudera/parcels/SPARK2/lib/spark2/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1133, in __call__
14 File "/opt/cloudera/parcels/SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354/lib/spark2/python/lib/pyspark.zip/pyspark/sql/utils.py", line 79, in deco
15 pyspark.sql.utils.IllegalArgumentException: u"requirement failed: The number of columns doesn't match.\nOld column names (2): Continent, Country\nNew column names (0): "

Changes

 

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


In this Document
Symptoms
Changes
Cause
Solution
References


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.