My Oracle Support Banner

Node 3 Migration Fails on "InstallHadoop" Because the MySQL scm User for the Migrated Host Has Not Been Granted the Required Permissions (Doc ID 2524102.1)

Last updated on NOVEMBER 08, 2019

Applies to:

Big Data Appliance Integrated Software - Version 4.11.0 and later
Linux x86-64

Symptoms

Note: In the example here Node 3 is being migrated to Node 5 on BDA 4.11.  In this case Node 3 is the default BDA host with Cloudera Manager and MySQL Master role.

The symptoms include:

1. Node 3 migration repeatedly fails on the "InstallHadoop" Step with:

************************************
Error [24192]: (//<HOSTNAME5>.<DOMAIN>//Stage[main]/Hadoop::Installpkg2/Exec[enablecmhttps_script]/returns) change from notrun to 0 failed: /opt/oracle/BDAMammoth/bdaconfig/tmp/enablecmhttps.sh &>
/opt/oracle/BDAMammoth/bdaconfig/tmp/enablecmhttps_<##>.out
returned 1 instead of one of [0]
************************************

a) The /opt/oracle/BDAMammoth/bdaconfig/tmp/enablecmhttps_<##>.out file on <HOSTNAME5> shows output like below. Likely because
<HOSTNAME5> is down during migration.

Failed. Full error log in : /opt/oracle/BDAMammoth/bdaconfig/tmp/cm_config_<##>.err
json-select: no valid input found for "items?s(name:"KEYSTORE_PATH")/value/@bash@"
json-select: no valid input found for "items?s(name:"KEYSTORE_PASSWORD")/value/@bash@"

b) The /opt/oracle/BDAMammoth/bdaconfig/tmp/cm_config_<##>.err file shows a failed curl command to Cloudera Manager on <HOSTNAME5> because it is not up at this point.

2. The MySQL Master on the node being migrated to is up and looks healthy, and the MySQL Slave is up and healthy as well. MySQL replication is on.

3. Cloudera Manager (CM) is not up because the CM server fails to start on the host where <HOSTNAME3> is being migrated to.

The CM agent logs, /var/log/cloudera-scm-agent/cloudera-scm-agent.log, show errors like below:

<TIMESTAMP1> WARN com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1:com.mchange.v2.resourcepool.BasicResourcePool: com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@=<###> -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (5). Last acquisition attempt exception:
java.sql.SQLException: Access denied for user 'scm'@'<HOSTNAME5>.<DOMAIN>' (using password: YES)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:996)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3887)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3823)
...
<TIMESTAMP3> ERROR main:org.hibernate.engine.jdbc.spi.SqlExceptionHelper: Connections could not be acquired from the underlying database!
...
<TIMESTAMP4> ERROR main:com.cloudera.server.cmf.Main: Server failed.
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'com.cloudera.server.cmf.TrialState': Cannot resolve reference to bean 'entityManagerFactoryBean' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactoryBean': FactoryBean threw exception on object creation; nested exception is javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: Could not open connection
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:328)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:106)
at org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:616)
at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:148)
...
Caused by: java.sql.SQLException: Connections could not be acquired from the underlying database!
at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:106)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:529)

 

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


In this Document
Symptoms
Cause
Solution


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.