BDA Rolling Upgrade to BDA V4.3 Fails on Step 5 - Failed to upgrade cluster - Parcel is not Upgraded
(Doc ID 2081795.1)
Last updated on MAY 18, 2018
Applies to:Big Data Appliance Integrated Software - Version 4.3.0 and later
BDA rolling upgrade to BDA V4.3 from BDA V4.2 fails on Step 5. The CDH parcel is not upgraded, upgrade fails. For details on rolling upgrade see: Upgrading Oracle Big Data Appliance Cluster to V4.3 with Mammoth (Software Deployment Bundle) Release V4.3 Frequently Asked Questions (FAQ) (Doc ID 2080984.1).
The symptoms for the Step 5 rolling upgrade parcel failure are as below:
1. On a rolling upgrade to BDA V4.3 from BDA V4.2 Step 5 of the upgrade fails with:
Error : (//bdanode03.example.com//Stage[main]/Hadoop::Installcdhparcel/Exec[deployparcel]/returns)
change from notrun to 0 failed: /opt/oracle/BDAMammoth/bdaconfig/tmp/deployparcel.sh &>
/opt/oracle/BDAMammoth/bdaconfig/tmp/deployparcel_1447900904.out returned 1 instead of one of 
2. The /opt/oracle/BDAMammoth/bdaconfig/tmp/deployparcel_1447900904.out shows failures related to the parcel upgrade. Possible failure output can look like below. Different failures are possible.
a) Example output:
Result Message is: "Failed to upgrade cluster"
b) Example output:
Failed. Full error output in : /opt/oracle/BDAMammoth/bdaconfig/tmp/clusters_neoscl_commands_upgradeCdh.out
API Version used is v10
Where the associated /opt/oracle/BDAMammoth/bdaconfig/tmp/clusters_neoscl_commands_upgradeCdh.out shows:
3. Some of the parcel upgrade is in place:
a) 'dcli -C hadoop version' reports "Hadoop 2.6.0-cdh5.4.7" on each node.
b) From Cloudera Manager (CM), navigating to Home > <cluster_name> > shows the expected: CDH 5.4.7 Parcels.
4. However from CM, navigating to Hosts > Parcels > Parcel Usage, shows that the parcel upgrade is incomplete on one or more hosts. One or more hosts report a parcel status of:
There are nodes in the cluster where the parcel upgrade is not complete. Although parcel upgrade may have been completed on some of the nodes as well.
a) Example of 3 nodes with "Multiple product versions running on a single host":
b) Example of 6 nodes with "Multiple product versions running on a single host":
5. In CM navigating to > All Recent Commands shows various errors indicating that the rolling upgrade failed. Failures look like:
a) Failed to perform Rolling Restart on the cluster
b) Failed to execute command Rolling restart on service hdfs
c) Failed to perform Rolling Restart
d) Execute command RollingRestart on cluster <cluster_name>
e) Failed to upgrade cluster
a) Sample errors:
b) Sample errors:
c) Sample errors:
d) There may be additional messaging about waiting for journal nodes to get in sync:
6. Navigating to the CM Home page shows that the hdfs is in bad health due to both NameNodes being down. See:
7. Other services may be down as well, for example yarn, zookeeper, HBase.
To view full details, sign in with your My Oracle Support account.
Don't have a My Oracle Support account? Click to get started!