After BDA CDH Upgrade to V4.* from V3.* the Navigator Audit Server is Down Due to Incorrect MySQL Java Connector

(Doc ID 1934609.1)

Last updated on JUNE 01, 2016

Applies to:

Big Data Appliance Integrated Software - Version 3.0.1 and later
Linux x86-64

Symptoms

After BDA CDH Upgrade to V4.* from V3.* the Navigator Audit Server is Down.  This can however happen when installing Navigator V3.1.

Symptoms looks like:

1. Cloudera Manager shows a Critical Health Issue and the Cloudera Management Service is reported as "red"/"Bad Health":

2. Navigating to Home > mgmt (Cloudera Management Service) shows that the "Navigator Audit Server" is down and that "The Navigator Audit Server is not running.":

3. Additional output in the logs on the Navigator Audit Server host under: /var/log/cloudera-scm-navigator show Errors like:

a) The file: mgmt-cmf-mgmt-NAVIGATOR-bdaode04.example.com.log.out shows ERRORS like: "Unable to initialize audit server" due to JDBC driver class not found:

2014-10-08 15:23:15,290 ERROR com.cloudera.navigator.NavigatorMain: Unable to initialize audit server.
javax.persistence.PersistenceException: [PersistenceUnit: navigator] Unable to build EntityManagerFactory
at org.hibernate.ejb.Ejb3Configuration.buildEntityManagerFactory(Ejb3Configuration.java:924)
at org.hibernate.ejb.Ejb3Configuration.buildEntityManagerFactory(Ejb3Configuration.java:899)
at org.hibernate.ejb.HibernatePersistence.createEntityManagerFactory(HibernatePersistence.java:59)
at javax.persistence.Persistence.createEntityManagerFactory(Persistence.java:63)
at com.cloudera.enterprise.CommonMain.setupHibernate(CommonMain.java:153)
at com.cloudera.navigator.NavigatorServer.<init>(NavigatorServer.java:76)
at com.cloudera.navigator.NavigatorMain.main(NavigatorMain.java:20)

Caused by: org.hibernate.service.classloading.spi.ClassLoadingException: HHH010003: JDBC Driver class not found: com.mysql.jdbc.Driver
        at org.hibernate.service.jdbc.connections.internal.C3P0ConnectionProvider.configure(C3P0ConnectionProvider.java:142)

and

2014-10-08 15:24:21,850 ERROR com.cloudera.navigator.NavigatorMain: Unable to initialize audit server.
...

Caused by: java.lang.ClassNotFoundException: Could not load requested class : com.mysql.jdbc.Driver
        at org.hibernate.service.classloading.internal.ClassLoaderServiceImpl$AggregatedClassLoader.findClass(ClassLoaderServiceImpl.java:296)
...

 

b) The Navigator Metastore is logging problems setting up the extraction task for HDFS.  See the file: mgmt-cmf-mgmt-NAVIGATORMETASERVER-bdanode04.example.com.log.out  and find various Errors like: "Error extracting Oozie workflows" and "ExecutionException running extraction tasks for service for hdfs".

2014-10-08 15:23:40,985 ERROR com.cloudera.nav.oozie.extractor.OozieExtractor: Error extracting Oozie workflows.
IO_ERROR : java.net.ConnectException: Connection refused
at org.apache.oozie.client.OozieClient.validateWSVersion(OozieClient.java:312)
at org.apache.oozie.client.OozieClient.createURL(OozieClient.java:395)
at org.apache.oozie.client.OozieClient.access$000(OozieClient.java:76)
...
at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:186)
at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:232)
at org.apache.oozie.client.AuthOozieClient.createConnection(AuthOozieClient.java:128)
...
2014-10-08 15:24:41,023 INFO com.cloudera.cmf.cdhclient.CdhExecutorFactory: Instantiating CDH classloader: version CDH5, key b4fe448702cb0fc3eacda61b4ce16558, path /usr/share/cmf/cloudera-navigator-server/libs/cdh5
2014-10-08 15:24:41,023 INFO com.cloudera.cmf.cdhclient.CdhExecutorFactory: Instantiating CDH classloader: version CDH5, key cc9370a975e3b182c53c30987a7dc204, path /usr/share/cmf/cloudera-navigator-server/libs/cdh5

and

2014-10-08 22:26:26,858 WARN com.cloudera.nav.extract.CdhClientExtractionTask: ExecutionException running extraction tasks for service '<cluster-name>::hdfs'.
2014-10-08 22:26:26,858 WARN com.cloudera.nav.extract.CdhClientExtractionTask: Stack trace:
java.util.concurrent.ExecutionException: java.lang.IllegalStateException
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:188)

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms