Upgrade to Mammoth 4.4.0 or higher Fails with "ERROR: Upgrade Cloudera Manager agents on off BDA client hosts before continue with Upgrade" (Doc ID 2134275.1)

Last updated on NOVEMBER 10, 2016

Applies to:

Big Data Appliance Integrated Software - Version 4.4.0 and later
Linux x86-64

Symptoms

The symptoms are:

1. Upgrading a multi-rack BDA cluster from 4.2.0 to 4.4.0 raises the following error at step 1:

INFO: Puppet agent run on node <rack1>node18 completed successfully.
INFO: Check the log file in /opt/oracle/BDAMammoth/bdaconfig/tmp/pagent-<rack1>node18-20160429122606.log
ERROR: Upgrade Cloudera Manager agents on off BDA client hosts before continue with Upgrade
Exiting...

2. Errors like below are found in the pagent-<rack1>node18-20160429122606.log:

Apr 29 12:26:27 <rack1>node01 puppet-master[10841]: Compiled catalog for <rack1>node18.example.com in environment production in 0.47 seconds
Apr 29 12:26:27 <rack1>node01 puppet-master[10841]: (//<rack1>node18.example.com//Stage[main]/Hadoop::Offbdahostsprecheck/File[offbda_hosts_precheck]/ensure) created
Apr 29 12:26:27 <rack1>node01 puppet-master[10841]: (//<rack1>node18.example.com/Puppet) Finished catalog run in 0.16 seconds

3. Following document "On Oracle Big Data Appliance Upgrading to 4.4 Release, Fails at Step1 when SSL is Enabled for Cloudera \Manager(CM) (Doc ID 2111784.1)"does not help.  The problem is not the same.  In this case it is not possible to find:

a) /opt/oracle/BDAMammoth/bdaconfig/tmp/offbdahostsprecheck.out file to find the link for https://<BDANode03>:7183/cmf/command/<id>/download.

or

b) /opt/oracle/BDAMammoth/bdaconfig/tmp/hostInspectorResultData.out on node01.

Following the step in 2111784.1 to create a file from the Cloudera Manager Host Inspector and copy it to  the Mammoth node (node 1) to: /opt/oracle/BDAMammoth/bdaconfig/tmp/hostInspectorResultData.out does not help.  The same error is raised when rerunning the upgrade:

INFO: Puppet agent run on node <rack1>node18 completed successfully.
INFO: Check the log file in /opt/oracle/BDAMammoth/bdaconfig/tmp/pagent-<rack1>node18-20160429170144.log
ERROR: Upgrade Cloudera Manager agents on off BDA client hosts before continue with Upgrade
Exiting...

 

4. Check for the /opt/oracle/BDAMammoth/bdaconfig/tmp/offhostsprecheck.out file on the NameNode.

a) To find the NameNode check for /opt/oracle/BDAMammoth/puppet/manifests/environment.pp from node1 and look for '$namenode_node'.

For example:

$namenode_node = "<rack2>node01.example.com"

Note: This may not show the active NameNode. There are 2 NameNodes and one is <rack2>node01 and one is <rack1>node02 in multi-rack cluster. At any given time either NameNode may be active and the code makes no assumptions about which NameNode is active - $namenode_node is not intended to always be the active NameNode.

b) Check in Cloudera Manager to find the active/standby NameNodes.

c) Running 'offbdahostsprecheck.sh' manually shows the following:

# ./offbdahostsprecheck.sh Nodes managed by mammoth = 18 API Version used is v10 Succeeded. Output in : /opt/oracle/BDAMammoth/bdaconfig/tmp/clusters_<cluster>_hosts_1461972281.out
Nodes managed by CM = 72
There are Off BDA hosts installed on the cluster API Version used is v10 Succeeded. Output in : /opt/oracle/BDAMammoth/bdaconfig/tmp/clusters_<cluster>_commands_inspectHosts.out
Command ID is 54559
.....
Command 54559 finished after 30 seconds
Operation completed successfully
API Version used is v10
Succeeded. Output in : /opt/oracle/BDAMammoth/bdaconfig/tmp/commands_54559.out
Results in
/opt/oracle/bda/bin/json-select: "/opt/oracle/BDAMammoth/bdaconfig/tmp/hostInspectorResultData.out" No such file.
Agents are not updated

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms