Hive Metastore and Webhcat Test Failures During BDA Cluster Verifications (./mammoth -c) After Upgrade to V4.5. (Doc ID 2265111.1)

Last updated on JUNE 09, 2017

Applies to:

Big Data Appliance Integrated Software - Version 4.5.0 and later
Linux x86-64

Symptoms

On Oracle Big Data Appliance after upgrade to BDA V4.5 the Hive Metastore and Webhcat tests fail when running the cluster verifications with "./mammoth -c". 

The following errors are seen:

...

ERROR: Errors in Metastore test run
ERROR: Errors in Webhcat test run
INFO: Executing oracleUserDestroy.sh on nodes /opt/oracle/BDAMammoth/bdaconfig/tmp/all_nodes #Step -1#
SUCCESS: Executed oracleUserDestroy.sh on nodes /opt/oracle/BDAMammoth/bdaconfig/tmp/all_nodes #Step -1#
INFO: Executing remTestUsers.sh on nodes bdanode01 #Step -1#
SUCCESS: Executed remTestUsers.sh on nodes bdanode01 #Step -1#
SUCCESS: Successfully removed Kerberos test users.
SUCCESS: Ran /bin/cp -pr /opt/oracle/BDAMammoth/bdaconfig/tmp/* /opt/oracle/bda/install/log/clusterchk/summary-<timestamp> and it returned: RC=0
SUCCESS: Ran /bin/rm -rf /opt/oracle/BDAMammoth/bdaconfig/tmp/* and it returned: RC=0
SUCCESS: Ran /bin/cp -prf /tmp/bdacheckcluster* /opt/oracle/bda/install/log/clusterchk/summary-<timestamp> and it returned: RC=0
INFO: Install summary copied to /opt/oracle/bda/install/log/clusterchk/summary-<timestamp>
ERROR: Errors found in cluster validation checks
ERROR: Resolve the errors and then re-run mammoth with the -c option
INFO: Time spent in post-cleanup operations is 757 seconds.
==================================================================================
ERROR: Cluster checks failed!
ERROR: Resolve the errors and then re-run mammoth with the -c option

 

Additional failure output may look like below failing with "User oracle does not have privileges for CREATEDATABASE".

Webhcat Test output
------------------------------------------------------------------------------
TEST: create a hcatlog database directory
TEST: create a hcatlog database
{"error":"FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:User oracle does not have privileges for CREATEDATABASE)","errorDetail":"org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:User oracle does not have privileges for CREATEDATABASE)\n\tat org.apache.hadoop.hive.ql.metadata.Hive.createDatabase(Hive.java:347)\n\tat org.apache.hadoop.hive.ql.exec.DDLTask.createDatabase(DDLTask.java:3796)\n\tat org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:267)\n\tat org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)\n\tat org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88)\n\tat org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1636)\n\tat org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1396)\n\tat org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1181)\n\tat org.apache.hadoop.hive.ql.Driver.run(Driver.java:1047)\n\tat org.apache.hadoop.hive.ql.Driver.run(Driver.java:1037)\n\tat org.apache.hive.hcatalog.cli.HCatDriver.run(HCatDriver.java:43)\n\tat org.apache.hive.hcatalog.cli.HCatCli.processCmd(HCatCli.java:291)\n\tat org.apache.hive.hcatalog.cli.HCatCli.processLine(HCatCli.java:245)\n\tat org.apache.hive.hcatalog.cli.HCatCli.main(HCatCli.java:183)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:497)\n\tat org.apache.hadoop.util.RunJar.run(RunJar.java:221)\n\tat org.apache.hadoop.util.RunJar.main(RunJar.java:136)\nCaused by: MetaException(message:User oracle does not have privileges for CREATEDATABASE)\n\tat org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_database_result$create_database_resultStandardScheme.read(ThriftHiveMetastore.java:14326)\n\tat org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_database_result$create_database_resultStandardScheme.read(ThriftHiveMetastore.java:14294)\n\tat org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_database_result.read(ThriftHiveMetastore.java:14228)\n\tat org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)\n\tat org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_create_database(ThriftHiveMetastore.java:621)\n\tat org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.create_database(ThriftHiveMetastore.java:608)\n\tat org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createDatabase(HiveMetaStoreClient.java:645)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:497)\n\tat org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:91)\n\tat com.sun.proxy.$Proxy5.createDatabase(Unknown Source)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:497)\n\tat org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:1966)\n\tat com.sun.proxy.$Proxy5.createDatabase(Unknown Source)\n\tat org.apache.hadoop.hive.ql.metadata.Hive.createDatabase(Hive.java:341)\n\t... 19 more\n","errorCode":40000,"sqlState":"08S01","database":"webhcat_test_db_1496955625"}ERROR: Timed out running creating a hcatlog database. HTTP/1.1 500 Server Error
Please check details in webhcat_test.out

{"error":"FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:User oracle does not have privileges for CREATEDATABASE)","errorDetail":"org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:User oracle does not have privileges for CREATEDATABASE)\n\tat org.apache.hadoop.hive.ql.metadata.Hive.createDatabase(Hive.java:347)\n\tat org.apache.hadoop.hive.ql.exec.DDLTask.createDatabase(DDLTask.java:3796)\n\tat org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:267)\n\tat org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)\n\tat org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88)\n\tat org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1636)\n\tat org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1396)\n\tat org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1181)\n\tat org.apache.hadoop.hive.ql.Driver.run(Driver.java:1047)\n\tat org.apache.hadoop.hive.ql.Driver.run(Driver.java:1037)\n\tat org.apache.hive.hcatalog.cli.HCatDriver.run(HCatDriver.java:43)\n\tat org.apache.hive.hcatalog.cli.HCatCli.processCmd(HCatCli.java:291)\n\tat org.apache.hive.hcatalog.cli.HCatCli.processLine(HCatCli.java:245)\n\tat org.apache.hive.hcatalog.cli.HCatCli.main(HCatCli.java:183)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:497)\n\tat org.apache.hadoop.util.RunJar.run(RunJar.java:221)\n\tat org.apache.hadoop.util.RunJar.main(RunJar.java:136)\nCaused by: MetaException(message:User oracle does not have privileges for CREATEDATABASE)\n\tat org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_database_result$create_database_resultStandardScheme.read(ThriftHiveMetastore.java:14326)\n\tat org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_database_result$create_database_resultStandardScheme.read(ThriftHiveMetastore.java:14294)\n\tat org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_database_result.read(ThriftHiveMetastore.java:14228)\n\tat org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)\n\tat org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_create_database(ThriftHiveMetastore.java:621)\n\tat org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.create_database(ThriftHiveMetastore.java:608)\n\tat org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createDatabase(HiveMetaStoreClient.java:645)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:497)\n\tat org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:91)\n\tat com.sun.proxy.$Proxy5.createDatabase(Unknown Source)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:497)\n\tat org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:1966)\n\tat com.sun.proxy.$Proxy5.createDatabase(Unknown Source)\n\tat org.apache.hadoop.hive.ql.metadata.Hive.createDatabase(Hive.java:341)\n\t... 19 more\n","errorCode":40000,"sqlState":"08S01","database":"webhcat_test_db_1496955625"}ERROR: Timed out running creating a hcatlog database. HTTP/1.1 500 Server Error
Webhcat test failed

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms