My Oracle Support Banner

HiveServer2 Concurrency is Not Working Even Though the HS2 Concurrency Flag is Set "message:javax.jdo.JDODataStoreException" Exceptions are Raised (Doc ID 2146928.1)

Last updated on JULY 26, 2021

Applies to:

Big Data Appliance Integrated Software - Version 4.1.0 and later
Linux x86-64

Symptoms

NOTE: In the examples that follow, user details, cluster names, hostnames, directory paths, filenames, etc. represent a fictitious sample (and are used to provide an illustrative example only). Any similarity to actual persons, or entities, living or dead, is purely coincidental and not intended in any manner. 

HiveServer2 concurrency does not work even though the concurrency flag is set for HS2.  The scenario is:

1. The HS2 concurrency flag is set:

<property>
<name>hive.support.concurrency</name>
<value>true</value>
</property>
<property>
<name>hive.zookeeper.quorum</name>
<value>bdanode01.example.com,bdanode02.example.com,bdanode03.example.com</value>
</property>


2. There are multiple concurrent/parallel beeline jobs copying data from staging tables to the same final Hive managed table. There are times when all of the beeline sessions/jobs are trying to insert into the table at the same time.

3. However the following failures are observed:

a)  The Metastore fails with: MetaException(message:javax.jdo.JDODataStoreException: Insert of object:

1 ERROR org.apache.hadoop.hive.ql.exec.Task: Failed with exception MetaException(message:javax.jdo.JDODataStoreException: Insert of object  "org.apache.hadoop.hive.metastore.model.X@Y" using statement  "INSERT INTO `PARTITIONS` (`X`,`Y`,`Z`,`Z1`,`A`,`B`)  VALUES (?,?,?,?,?,?)" failed : Duplicate entry 'as_of_date=2016-04-26-230716' for key 'X'
1 ERROR org.apache.hadoop.hive.ql.exec.Task: Error during job, obtaining debugging information...



b) The Hiveserver2.log shows:

2016-04-26 08:36:40,816 ERROR hive.ql.metadata.Hive: MetaException(message:javax.jdo.JDODataStoreException:
Insert of object "org.apache.hadoop.hive.metastore.model.X@Y using statement "INSERT INTO `PARTITIONS` (`X`,`Y`,`Z`,`A`,`B`,`C`)  VALUES (?,?,?,?,?,?)" failed : Duplicate entry 'as_of_date=2016-04-26-75871' for key 'X'
NestedThrowables:
com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Duplicate entry 'as_of_date=2016-04-26-75871' for key 'X')
  at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$append_partition_with_environment_context_result$append_partition_with_environment_context_resultStandardScheme.read(ThriftHiveMetastore.java:41079)
...

 

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!

In this Document
Symptoms
Cause
Solution


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.