My Oracle Support Banner

Upgrade from BDA v4.4 to v4.8 Fails at Step 3 with "ERROR: Ensure that HDFS is up and healthy before restarting the upgrade" Because the HDFS Service Will Not Start (Doc ID 2299407.1)

Last updated on JANUARY 15, 2020

Applies to:

Big Data Appliance Integrated Software - Version 4.4.0 and later
Linux x86-64

Symptoms

NOTE: In the examples that follow, user details, cluster names, hostnames, directory paths, filenames, etc. represent a fictitious sample (and are used to provide an illustrative example only). Any similarity to actual persons, or entities, living or dead, is purely coincidental and not intended in any manner. 

Step 3 of a Mammoth upgrade from 4.4 to 4.8 fails with "Ensure that HDFS is up and healthy before restarting the upgrade". 

1. Mammoth upgrade output reports the error message as below:

...

SUCCESS: Successfully copied JCE policy files.
INFO: Creating nodelist files...
INFO: Creating environment.pp file ...
INFO: Time spent in step 2 UpdateBaseImage is 1594 seconds.
INFO: Checking if password-less ssh is set up
INFO: Executing checkRoot.sh on nodes /opt/oracle/BDAMammoth/bdaconfig/tmp/all_nodes #Step 3#
SUCCESS: Executed checkRoot.sh on nodes /opt/oracle/BDAMammoth/bdaconfig/tmp/all_nodes #Step 3#
INFO: Executing checkSSHAllNodes.sh on nodes /opt/oracle/BDAMammoth/bdaconfig/tmp/all_nodes #Step 3#
SUCCESS: Executed checkSSHAllNodes.sh on nodes /opt/oracle/BDAMammoth/bdaconfig/tmp/all_nodes #Step 3#
INFO: Step 3: Changing oozie hdfs directory permissions
INFO: Value is not set for AD Account Prefix. Setting to blank.
SUCCESS: Successfully copied keytabs to Mammoth node.
INFO: Executing oracleUser.sh on nodes /opt/oracle/BDAMammoth/bdaconfig/tmp/all_nodes #Step 3#
SUCCESS: Executed oracleUser.sh on nodes /opt/oracle/BDAMammoth/bdaconfig/tmp/all_nodes #Step 3#
INFO: Executing hdfsUser.sh on nodes /opt/oracle/BDAMammoth/bdaconfig/tmp/all_nodes #Step 3#
SUCCESS: Executed hdfsUser.sh on nodes /opt/oracle/BDAMammoth/bdaconfig/tmp/all_nodes #Step 3#
ERROR: Ensure that HDFS is up and healthy before restarting the upgrade
ERROR: Cannot continue with operation
INFO: Time spent in step 3 StopClouderaManager is 2555 seconds.
INFO: Running bdadiagcluster...

2. Checking in Cloudera Manager shows that the HDFS service will not start.

3. Investigating the HDFS DataNode logs shows a security exception like below:

at com.cloudera.io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:982)
at com.cloudera.io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:908)
at com.cloudera.io.netty.handler.ssl.SslHandler.decode(SslHandler.java:854)
com.cloudera.io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: No appropriate protocol (protocol is disabled or cipher suites are inappropriate)
Caused by: javax.net.ssl.SSLHandshakeException: No appropriate protocol (protocol is disabled or cipher suites are inappropriate)
at sun.security.ssl.Handshaker.activate(Handshaker.java:503)
at sun.security.ssl.SSLEngineImpl.kickstartHandshake(SSLEngineImpl.java:729)
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:829)
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
at com.cloudera.io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:982)
at com.cloudera.io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:908)
at com.cloudera.io.netty.handler.ssl.SslHandler.decode(SslHandler.java:854)

 

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


In this Document
Symptoms
Cause
Solution
References


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.