Step 5 of BDA V4.4 upgrade Fails-Navigator Metadata Server is Down with "Access denied for user 'nmeta'@'bdanode04.example.com'" (Doc ID 2123359.1)

Last updated on APRIL 04, 2016

Applies to:

Big Data Appliance Integrated Software - Version 4.2.0 and later
Linux x86-64

Symptoms

Step 5 of a BDA V4.3 to V4.4 upgrade fails with the Navigator Metadata Server being down with "Access denied for user 'nmeta'@'bdanode04.example.com'".

Symptoms:

Step 5 of 4.3 > 4.4 upgrade fails because the Navigator Metadata Server is down. The error is:

Caused by: java.sql.SQLException: Access denied for user 'nmeta'@'bdanode04.example.com' (using password: YES)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:996)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3887)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3823)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:870)
at com.mysql.jdbc.MysqlIO.proceedHandshakeWithPluggableAuthentication(MysqlIO.java:1659)
at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1206)
at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2234)
at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2265)
at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2064)
at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:790)
at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:44)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessor

The Navigator Metadata Server log shows:

2016-04-01 18:27:20,242 ERROR com.cloudera.nav.server.NavServer [main]: Unable to access schema version in database
java.lang.RuntimeException: java.sql.SQLException: Access denied for user 'nmeta'@'bdanode04.example.com' (using password: YES)
at com.google.common.base.Throwables.propagate(Throwables.java:160)
at com.cloudera.nav.server.NavServer.initializeDatabase(NavServer.java:199)
at com.cloudera.nav.server.NavServer.run(NavServer.java:417)
at com.cloudera.nav.server.NavServer.main(NavServer.java:113)
Caused by: java.sql.SQLException: Access denied for user 'nmeta'@'bdanode04.example.com' (using password: YES)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:996)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3887)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3823)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:870)
at com.mysql.jdbc.MysqlIO.proceedHandshakeWithPluggableAuthentication(MysqlIO.java:1659)
at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1206)
at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2234)
at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2265)
at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2064)
at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:790)
at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:44)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:377)
at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:395)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:325)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at com.cloudera.enterprise.dbutil.DbUtil.upgradeSchema(DbUtil.java:394)
at com.cloudera.nav.server.NavServer.initializeDatabase(NavServer.java:195)
... 2 more

 

Troubleshooting:

1. Check the "History and Rollback" for Role config group 'Navigator Metadata Server Default Group' to see if it contains the old database credentials.

To reach the "History and Rollback" page in CM, navigate:
CM Home Page > mgmt Service > Configuration > click "History and Rollback" button.

In the case of these symptoms, "History and Rollback" contains old database credentials. The page shows output like:

Navigator Metadata Server Database Name: navigator_metadata
navms (scratched out)
default value (scratched out)

Navigator Metadata Server Database Username: nmeta
navms (scratched out)
default value (scratched out)

 

2. Check which user has the required privileges for the navigator databases using the below as an example.

a) Get the navigator database: 

In the example here, the navms user has access to the navms Navigator Metadata server database.  

4. Check the values in Cloudera Manager for:

  • Navigator Metadata Server Database Name navms.db.name
  • Navigator Metadata Server Database Username navms.db.user
  • Navigator Audit Server Database Name navigator.db.name
  • Navigator Audit Server Database Username navigator.db.user

a) In CM navigate:

Home > mgmt > Navigator Metadata Server > Configuration > Search: Database

Check the values for:

Navigator Metadata Server Database Name navms.db.name
Navigator Metadata Server Database Username navms.db.user

b) In CM navigate:

Home > mgmt > Navigator Audit Server > Configuration > Search: Database

Check the values for:

Navigator Audit Server Database Name navigator.db.name
Navigator Audit Server Database Username navigator.db.user

In the example here:

Navigator Metadata Server Database Name navms.db.name: navigator_metadata
Navigator Metadata Server Database Username navms.db.user: nmeta

and

Navigator Audit Server Database Name navigator.db.name: navigatorDB
Navigator Audit Server Database Username navigator.db.user: root

Observe that upgrade incorrectly changed the original database and user name for Navigator Metadata Server but correctly did not for Navigator Audit Server.

Background

1. On a cluster upgrade to BDA V4.4 where the original cluster did not support Navigator, the defaults installed by the Mammoth 4.4 upgrade are:

a) In CM:
Navigator Metadata Server Database Name navms.db.name: navigator_metadata
Navigator Metadata Server Database Username navms.db.user: nmeta

Navigator Audit Server Database Name navigator.db.name: navigator
Navigator Audit Server Database Username navigator.db.user: naudit

b) In MySql:

The navigator_metadata and navigator databases are created and nmeta and naudit users created respectively.

2. On a cluster upgrade to BDA V4.4 where the original cluster did support Navigator, the Mammoth 4.4 upgrade should not touch the existing values.

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms