BDS SQL Queries Using 100% CPU When HBase Can Not Connect to the Hive MetaStore.
(Doc ID 2209218.1)
Last updated on JANUARY 15, 2020
Applies to:
Oracle Big Data SQL - Version 3.0 and laterLinux x86-64
Symptoms
NOTE: In the examples that follow, user details, cluster names, hostnames, directory paths, filenames, etc. represent a fictitious sample (and are used to provide an illustrative example only). Any similarity to actual persons, or entities, living or dead, is purely coincidental and not intended in any manner.
BDS SQL Queries on the Exadata side are found to be consuming 100% of the system cpu.
The associated $ORACLE_HOME/bigdatasql/log/bigdata-log4j-<PID>.log shows a problem connecting to the Hive Metastore:
a) Output shows like:
[* describe.getInfo(6)] 16/11/14 17:11:15 INFO hive.metastore: Opened a connection to metastore, current connections: 1
[* describe.getInfo(6)] 16/11/14 17:11:15 INFO hive.metastore: Connected to metastore.
...
java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/mapreduce/TableInputFormatBase
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.hadoop.hive.hbase.HBaseStorageHandler.getInputFormatClass(HBaseStorageHandler.java:296)
at org.apache.hadoop.hive.ql.metadata.Table.getInputFormatClass(Table.java:302)
at org.apache.hadoop.hive.ql.exec.Utilities.getTableDesc(Utilities.java:1203)
at oracle.hadoop.sql.xcat.hive.XCatHiveInfo.getInfo(XCatHiveInfo.java:177)
at oracle.hadoop.sql.xcat.hive.XCatHive.getInfo(XCatHive.java:326)
...
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.mapreduce.TableInputFormatBase
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
...
[* describe.getInfo(6)] 16/11/14 17:11:15 INFO hive.metastore: Connected to metastore.
...
java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/mapreduce/TableInputFormatBase
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.hadoop.hive.hbase.HBaseStorageHandler.getInputFormatClass(HBaseStorageHandler.java:296)
at org.apache.hadoop.hive.ql.metadata.Table.getInputFormatClass(Table.java:302)
at org.apache.hadoop.hive.ql.exec.Utilities.getTableDesc(Utilities.java:1203)
at oracle.hadoop.sql.xcat.hive.XCatHiveInfo.getInfo(XCatHiveInfo.java:177)
at oracle.hadoop.sql.xcat.hive.XCatHive.getInfo(XCatHive.java:326)
...
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.mapreduce.TableInputFormatBase
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
...
b) Output also shows like:
[* describe.getInfo(6)] 16/11/14 17:11:27 INFO hive.metastore: Connected to metastore.
[* describe.getInfo(6)] 16/11/14 17:11:27 ERROR sql.JXADDescribe: JXADDescribe.getInfo
java.lang.NoClassDefFoundError: org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat
at org.apache.hadoop.hive.hbase.HBaseStorageHandler.getInputFormatClass(HBaseStorageHandler.java:296)
at org.apache.hadoop.hive.ql.metadata.Table.getInputFormatClass(Table.java:302)
at org.apache.hadoop.hive.ql.exec.Utilities.getTableDesc(Utilities.java:1203)
at oracle.hadoop.sql.xcat.hive.XCatHiveInfo.getInfo(XCatHiveInfo.java:177)
at oracle.hadoop.sql.xcat.hive.XCatHive.getInfo(XCatHive.java:326)
...
[* describe.getInfo(6)] 16/11/14 17:11:27 DEBUG metrics.TaskMetrics: jxad_task: TEST, time: 65, bytes: 0
[* describe.close] 16/11/14 17:11:27 DEBUG sql.TEST: Entering
[* describe.close] 16/11/14 17:11:27 INFO hive.metastore: Closed a connection to metastore, current connections: 0
[* describe.getInfo(6)] 16/11/14 17:11:27 ERROR sql.JXADDescribe: JXADDescribe.getInfo
java.lang.NoClassDefFoundError: org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat
at org.apache.hadoop.hive.hbase.HBaseStorageHandler.getInputFormatClass(HBaseStorageHandler.java:296)
at org.apache.hadoop.hive.ql.metadata.Table.getInputFormatClass(Table.java:302)
at org.apache.hadoop.hive.ql.exec.Utilities.getTableDesc(Utilities.java:1203)
at oracle.hadoop.sql.xcat.hive.XCatHiveInfo.getInfo(XCatHiveInfo.java:177)
at oracle.hadoop.sql.xcat.hive.XCatHive.getInfo(XCatHive.java:326)
...
[* describe.getInfo(6)] 16/11/14 17:11:27 DEBUG metrics.TaskMetrics: jxad_task: TEST, time: 65, bytes: 0
[* describe.close] 16/11/14 17:11:27 DEBUG sql.TEST: Entering
[* describe.close] 16/11/14 17:11:27 INFO hive.metastore: Closed a connection to metastore, current connections: 0
Cause
To view full details, sign in with your My Oracle Support account. |
|
Don't have a My Oracle Support account? Click to get started! |
In this Document
Symptoms |
Cause |
Solution |