My Oracle Support Banner

Improper Functioning of ODI 12c Spark To Hive KMs is Observed if the Target Hive Table is Partitioned (Doc ID 2367622.1)

Last updated on MARCH 02, 2018

Applies to:

Oracle Data Integrator - Version 12.1.3.0.0 and later
Information in this document applies to any platform.

Symptoms

The Oracle Data Integrator (ODI) 12c Spark To Hive Knowledge Modules will fail with following error if the target Hive table is partitioned:

Cannot convert the AST object to a task: Load FILTER_AP
Caused by: oracle.odi.domain.adapter.relational.IColumn.isUsedForPartitioning()Z at oracle.odi.mapping.generation.spark.SparkHiveStoreCmd.getPartitionColumns(SparkHiveStoreCmd.java:72)

The generated code from the Knowledge Module does not contain a partition clause. Analyzing the execution in simulation mode, it is possible to verify in the following section that the hive partitioned tables are not recognized:

Note: This is a sample code to demonstrate the issue and may vary according to each ODI project. 

# [oracle.odi.mapping.generation.spark.SparkHiveStoreCmd for table_tgt]
FLT = FLT.map(lambda FLT : {'some_id' : FLT.some_id,'realm_code' :
'#PROJECT_FOR_ORACLE_SUPPORT.V_PARTITION','dt' :
'#PROJECT_FOR_ORACLE_SUPPORT.V_SUBPARTITION'})
(sqlTypeName, inferSchemaMethod) = ('DataFrame', 'createDataFrame') if
sparkVersion >= 130 else ('SchemaRDD', 'inferSchema')
if sqlTypeName not in type(FLT).__name__ :
FLT = FLT.map(lambda row : Row(**row) if isinstance(row,dict) else row)
if FLT.take(1).__len__() > 0 :
FLT = getattr(hiveCtx, inferSchemaMethod)(FLT)
if sqlTypeName in type(FLT).__name__:
if sparkVersion > 141:
FLT.saveAsTable('m_shutkov.table_tgt', mode='overwrite')
else:
hiveCtx.sql('CREATE TABLE IF NOT EXISTS m_shutkov.table_tgt ( some_id
BIGINT , realm_code STRING , dt STRING )')
FLT.registerTempTable('HIVE_TMP_176')
hiveCtx.sql('INSERT OVERWRITE TABLE m_shutkov.table_tgt \
SELECT some_id , realm_code , dt FROM HIVE_TMP_176')

Changes

 

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.