BDS SQL Queries Using 100% CPU When HBase Can Not Connect to the Hive MetaStore.
(Doc ID 2209218.1)
Last updated on DECEMBER 02, 2016
Applies to:
Oracle Big Data SQL - Version 3.0 and laterLinux x86-64
Symptoms
BDS SQL Queries on the Exadata side are found to be consuming 100% of the system cpu.
The associated $ORACLE_HOME/bigdatasql/log/bigdata-log4j-<pid>.log shows a problem connecting to the Hive Metastore:
a) Output shows like:
[* describe.getInfo(6)] 16/11/14 17:11:15 INFO hive.metastore: Opened a connection to metastore, current connections: 1
[* describe.getInfo(6)] 16/11/14 17:11:15 INFO hive.metastore: Connected to metastore.
...
java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/mapreduce/TableInputFormatBase
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.hadoop.hive.hbase.HBaseStorageHandler.getInputFormatClass(HBaseStorageHandler.java:296)
at org.apache.hadoop.hive.ql.metadata.Table.getInputFormatClass(Table.java:302)
at org.apache.hadoop.hive.ql.exec.Utilities.getTableDesc(Utilities.java:1203)
at oracle.hadoop.sql.xcat.hive.XCatHiveInfo.getInfo(XCatHiveInfo.java:177)
at oracle.hadoop.sql.xcat.hive.XCatHive.getInfo(XCatHive.java:326)
...
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.mapreduce.TableInputFormatBase
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
...
[* describe.getInfo(6)] 16/11/14 17:11:15 INFO hive.metastore: Connected to metastore.
...
java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/mapreduce/TableInputFormatBase
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.hadoop.hive.hbase.HBaseStorageHandler.getInputFormatClass(HBaseStorageHandler.java:296)
at org.apache.hadoop.hive.ql.metadata.Table.getInputFormatClass(Table.java:302)
at org.apache.hadoop.hive.ql.exec.Utilities.getTableDesc(Utilities.java:1203)
at oracle.hadoop.sql.xcat.hive.XCatHiveInfo.getInfo(XCatHiveInfo.java:177)
at oracle.hadoop.sql.xcat.hive.XCatHive.getInfo(XCatHive.java:326)
...
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.mapreduce.TableInputFormatBase
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
...
b) Output also shows like:
[* describe.getInfo(6)] 16/11/14 17:11:27 INFO hive.metastore: Connected to metastore.
[* describe.getInfo(6)] 16/11/14 17:11:27 ERROR sql.JXADDescribe: JXADDescribe.getInfo
java.lang.NoClassDefFoundError: org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat
at org.apache.hadoop.hive.hbase.HBaseStorageHandler.getInputFormatClass(HBaseStorageHandler.java:296)
at org.apache.hadoop.hive.ql.metadata.Table.getInputFormatClass(Table.java:302)
at org.apache.hadoop.hive.ql.exec.Utilities.getTableDesc(Utilities.java:1203)
at oracle.hadoop.sql.xcat.hive.XCatHiveInfo.getInfo(XCatHiveInfo.java:177)
at oracle.hadoop.sql.xcat.hive.XCatHive.getInfo(XCatHive.java:326)
...
[* describe.getInfo(6)] 16/11/14 17:11:27 DEBUG metrics.TaskMetrics: jxad_task: TEST, time: 65, bytes: 0
[* describe.close] 16/11/14 17:11:27 DEBUG sql.TEST: Entering
[* describe.close] 16/11/14 17:11:27 INFO hive.metastore: Closed a connection to metastore, current connections: 0
[* describe.getInfo(6)] 16/11/14 17:11:27 ERROR sql.JXADDescribe: JXADDescribe.getInfo
java.lang.NoClassDefFoundError: org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat
at org.apache.hadoop.hive.hbase.HBaseStorageHandler.getInputFormatClass(HBaseStorageHandler.java:296)
at org.apache.hadoop.hive.ql.metadata.Table.getInputFormatClass(Table.java:302)
at org.apache.hadoop.hive.ql.exec.Utilities.getTableDesc(Utilities.java:1203)
at oracle.hadoop.sql.xcat.hive.XCatHiveInfo.getInfo(XCatHiveInfo.java:177)
at oracle.hadoop.sql.xcat.hive.XCatHive.getInfo(XCatHive.java:326)
...
[* describe.getInfo(6)] 16/11/14 17:11:27 DEBUG metrics.TaskMetrics: jxad_task: TEST, time: 65, bytes: 0
[* describe.close] 16/11/14 17:11:27 DEBUG sql.TEST: Entering
[* describe.close] 16/11/14 17:11:27 INFO hive.metastore: Closed a connection to metastore, current connections: 0
Cause
To view full details, sign in with your My Oracle Support account. |
|
Don't have a My Oracle Support account? Click to get started! |
In this Document
| Symptoms |
| Cause |
| Solution |