My Oracle Support Banner

Hive Metastore and Webhcat Test Failures During BDA Cluster Verifications (./mammoth -c) After Upgrade to V4.5. (Doc ID 2265111.1)

Last updated on JUNE 09, 2017

Applies to:

Big Data Appliance Integrated Software - Version 4.5.0 and later
Linux x86-64

Symptoms

On Oracle Big Data Appliance after upgrade to BDA V4.5 the Hive Metastore and Webhcat tests fail when running the cluster verifications with "./mammoth -c". 

The following errors are seen:

...

ERROR: Errors in Metastore test run
ERROR: Errors in Webhcat test run
INFO: Executing oracleUserDestroy.sh on nodes /opt/oracle/BDAMammoth/bdaconfig/tmp/all_nodes #Step -1#
SUCCESS: Executed oracleUserDestroy.sh on nodes /opt/oracle/BDAMammoth/bdaconfig/tmp/all_nodes #Step -1#
INFO: Executing remTestUsers.sh on nodes bdanode01 #Step -1#
SUCCESS: Executed remTestUsers.sh on nodes bdanode01 #Step -1#
SUCCESS: Successfully removed Kerberos test users.
SUCCESS: Ran /bin/cp -pr /opt/oracle/BDAMammoth/bdaconfig/tmp/* /opt/oracle/bda/install/log/clusterchk/summary-<timestamp> and it returned: RC=0
SUCCESS: Ran /bin/rm -rf /opt/oracle/BDAMammoth/bdaconfig/tmp/* and it returned: RC=0
SUCCESS: Ran /bin/cp -prf /tmp/bdacheckcluster* /opt/oracle/bda/install/log/clusterchk/summary-<timestamp> and it returned: RC=0
INFO: Install summary copied to /opt/oracle/bda/install/log/clusterchk/summary-<timestamp>
ERROR: Errors found in cluster validation checks
ERROR: Resolve the errors and then re-run mammoth with the -c option
INFO: Time spent in post-cleanup operations is 757 seconds.
==================================================================================
ERROR: Cluster checks failed!
ERROR: Resolve the errors and then re-run mammoth with the -c option

 

Additional failure output may look like below failing with "User oracle does not have privileges for CREATEDATABASE".

Webhcat Test output
------------------------------------------------------------------------------
TEST: create a hcatlog database directory
TEST: create a hcatlog database
{"error":"FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:User oracle does not have privileges for CREATEDATABASE)","errorDetail":"org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:User oracle does not have privileges for CREATEDATABASE)\n\tat org.apache.hadoop.hive.ql.metadata.Hive.createDatabase(Hive.java:347)\n\tat org.apache.hadoop.hive.ql.exec.DDLTask.createDatabase(DDLTask.java:3796)\n\tat org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:267)\n\tat org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)\n\tat org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88)\n\tat org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1636)\n\tat org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1396)\n\tat org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1181)\n\tat org.apache.hadoop.hive.ql.Driver.run(Driver.java:1047)\n\tat org.apache.hadoop.hive.ql.Driver.run(Driver.java:1037)\n\tat org.apache.hive.hcatalog.cli.HCatDriver.run(HCatDriver.java:43)\n\tat org.apache.hive.hcatalog.cli.HCatCli.processCmd(HCatCli.java:291)\n\tat org.apache.hive.hcatalog.cli.HCatCli.processLine(HCatCli.java:245)\n\tat org.apache.hive.hcatalog.cli.HCatCli.main(HCatCli.java:183)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:497)\n\tat org.apache.hadoop.util.RunJar.run(RunJar.java:221)\n\tat org.apache.hadoop.util.RunJar.main(RunJar.java:136)\nCaused by: MetaException(message:User oracle does not have privileges for CREATEDATABASE)\n\tat org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_database_result$create_database_resultStandardScheme.read(ThriftHiveMetastore.java:14326)\n\tat org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_database_result$create_database_resultStandardScheme.read(ThriftHiveMetastore.java:14294)\n\tat org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_database_result.read(ThriftHiveMetastore.java:14228)\n\tat org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)\n\tat org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_create_database(ThriftHiveMetastore.java:621)\n\tat org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.create_database(ThriftHiveMetastore.java:608)\n\tat org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createDatabase(HiveMetaStoreClient.java:645)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:497)\n\tat org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:91)\n\tat com.sun.proxy.$Proxy5.createDatabase(Unknown Source)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:497)\n\tat org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:1966)\n\tat com.sun.proxy.$Proxy5.createDatabase(Unknown Source)\n\tat org.apache.hadoop.hive.ql.metadata.Hive.createDatabase(Hive.java:341)\n\t... 19 more\n","errorCode":40000,"sqlState":"08S01","database":"webhcat_test_db_1496955625"}ERROR: Timed out running creating a hcatlog database. HTTP/1.1 500 Server Error
Please check details in webhcat_test.out

{"error":"FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:User oracle does not have privileges for CREATEDATABASE)","errorDetail":"org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:User oracle does not have privileges for CREATEDATABASE)\n\tat org.apache.hadoop.hive.ql.metadata.Hive.createDatabase(Hive.java:347)\n\tat org.apache.hadoop.hive.ql.exec.DDLTask.createDatabase(DDLTask.java:3796)\n\tat org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:267)\n\tat org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)\n\tat org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88)\n\tat org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1636)\n\tat org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1396)\n\tat org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1181)\n\tat org.apache.hadoop.hive.ql.Driver.run(Driver.java:1047)\n\tat org.apache.hadoop.hive.ql.Driver.run(Driver.java:1037)\n\tat org.apache.hive.hcatalog.cli.HCatDriver.run(HCatDriver.java:43)\n\tat org.apache.hive.hcatalog.cli.HCatCli.processCmd(HCatCli.java:291)\n\tat org.apache.hive.hcatalog.cli.HCatCli.processLine(HCatCli.java:245)\n\tat org.apache.hive.hcatalog.cli.HCatCli.main(HCatCli.java:183)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:497)\n\tat org.apache.hadoop.util.RunJar.run(RunJar.java:221)\n\tat org.apache.hadoop.util.RunJar.main(RunJar.java:136)\nCaused by: MetaException(message:User oracle does not have privileges for CREATEDATABASE)\n\tat org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_database_result$create_database_resultStandardScheme.read(ThriftHiveMetastore.java:14326)\n\tat org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_database_result$create_database_resultStandardScheme.read(ThriftHiveMetastore.java:14294)\n\tat org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_database_result.read(ThriftHiveMetastore.java:14228)\n\tat org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)\n\tat org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_create_database(ThriftHiveMetastore.java:621)\n\tat org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.create_database(ThriftHiveMetastore.java:608)\n\tat org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createDatabase(HiveMetaStoreClient.java:645)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:497)\n\tat org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:91)\n\tat com.sun.proxy.$Proxy5.createDatabase(Unknown Source)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:497)\n\tat org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:1966)\n\tat com.sun.proxy.$Proxy5.createDatabase(Unknown Source)\n\tat org.apache.hadoop.hive.ql.metadata.Hive.createDatabase(Hive.java:341)\n\t... 19 more\n","errorCode":40000,"sqlState":"08S01","database":"webhcat_test_db_1496955625"}ERROR: Timed out running creating a hcatlog database. HTTP/1.1 500 Server Error
Webhcat test failed

 

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!

In this Document
Symptoms
Cause
Solution
 Cluster with Network Encryption Disabled
 Cluster with Network Encryption Enabled

My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.