Upgrade from BDA v4.4 to v4.8 Fails at Step 3 with "ERROR: Ensure that HDFS is up and healthy before restarting the upgrade" Because the HDFS Service Will Not Start

(Doc ID 2299407.1)

Last updated on SEPTEMBER 03, 2017

Applies to:

Big Data Appliance Integrated Software - Version 4.4.0 and later
Linux x86-64

Symptoms

Step 3 of a Mammoth upgrade from 4.4 to 4.8 fails with "Ensure that HDFS is up and healthy before restarting the upgrade". 

1. Mammoth upgrade output reports the error message as below:

...

SUCCESS: Successfully copied JCE policy files.
INFO: Creating nodelist files...
INFO: Creating environment.pp file ...
INFO: Time spent in step 2 UpdateBaseImage is 1594 seconds.
INFO: Checking if password-less ssh is set up
INFO: Executing checkRoot.sh on nodes /opt/oracle/BDAMammoth/bdaconfig/tmp/all_nodes #Step 3#
SUCCESS: Executed checkRoot.sh on nodes /opt/oracle/BDAMammoth/bdaconfig/tmp/all_nodes #Step 3#
INFO: Executing checkSSHAllNodes.sh on nodes /opt/oracle/BDAMammoth/bdaconfig/tmp/all_nodes #Step 3#
SUCCESS: Executed checkSSHAllNodes.sh on nodes /opt/oracle/BDAMammoth/bdaconfig/tmp/all_nodes #Step 3#
INFO: Step 3: Changing oozie hdfs directory permissions
INFO: Value is not set for AD Account Prefix. Setting to blank.
SUCCESS: Successfully copied keytabs to Mammoth node.
INFO: Executing oracleUser.sh on nodes /opt/oracle/BDAMammoth/bdaconfig/tmp/all_nodes #Step 3#
SUCCESS: Executed oracleUser.sh on nodes /opt/oracle/BDAMammoth/bdaconfig/tmp/all_nodes #Step 3#
INFO: Executing hdfsUser.sh on nodes /opt/oracle/BDAMammoth/bdaconfig/tmp/all_nodes #Step 3#
SUCCESS: Executed hdfsUser.sh on nodes /opt/oracle/BDAMammoth/bdaconfig/tmp/all_nodes #Step 3#
ERROR: Ensure that HDFS is up and healthy before restarting the upgrade
ERROR: Cannot continue with operation
INFO: Time spent in step 3 StopClouderaManager is 2555 seconds.
INFO: Running bdadiagcluster...

2. Checking in Cloudera Manager shows that the HDFS service will not start.

3. Investigating the HDFS DataNode logs shows a security exception like below:

at com.cloudera.io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:982)
at com.cloudera.io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:908)
at com.cloudera.io.netty.handler.ssl.SslHandler.decode(SslHandler.java:854)
com.cloudera.io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: No appropriate protocol (protocol is disabled or cipher suites are inappropriate)
Caused by: javax.net.ssl.SSLHandshakeException: No appropriate protocol (protocol is disabled or cipher suites are inappropriate)
at sun.security.ssl.Handshaker.activate(Handshaker.java:503)
at sun.security.ssl.SSLEngineImpl.kickstartHandshake(SSLEngineImpl.java:729)
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:829)
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
at com.cloudera.io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:982)
at com.cloudera.io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:908)
at com.cloudera.io.netty.handler.ssl.SslHandler.decode(SslHandler.java:854)

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms