Upgrading Oracle Big Data Appliance From 4.2.0 to 4.5.0 Fails with "ERROR: Script could not be sent to some nodes" (Doc ID 2148194.1)

Last updated on AUGUST 30, 2017

Applies to:

Big Data Appliance Integrated Software - Version 4.5.0 and later
Linux x86-64

Symptoms

Upgrading Oracle Big Data Appliance (BDA) to 4.5.0 from 4.2.0 fails with the errors below on Step 3:

ERROR: Sending puppet script to node bdanode04 returned error 3
Warning: Setting manifestdir is deprecated. See http://links.puppetlabs.com/env-settings-deprecations
(at /usr/lib/ruby/site_ruby/1.8/puppet/settings.rb:1095:in 'issue_deprecations')
Warning: Setting modulepath is deprecated in puppet.conf. See http://links.puppetlabs.com/env-settings-deprecations
(at /usr/lib/ruby/site_ruby/1.8/puppet/settings.rb:1095:in 'issue_deprecations')
Warning: Puppet kick is deprecated. See http://links.puppetlabs.com/puppet-kick-deprecation
Warning: Failed to load ruby LDAP library. LDAP functionality will not be available
Triggering bda9node04.infoftps.com
Getting status
status is running
Host bdanode04.example.com is already running
bdanode04.example.com finished with exit code 3
Failed: bdanode04.example.comINFO: Puppet script successfully sent to node bdanode05
INFO: Puppet script successfully sent to node bdanode06
INFO: Puppet script successfully sent to node bdanode07
INFO: Puppet script successfully sent to node bdanode08
INFO: Puppet script successfully sent to node bdanode09
INFO: Puppet script successfully sent to node bdanode10
INFO: Puppet script successfully sent to node bdanode11
INFO: Puppet script successfully sent to node bdanode12
INFO: Puppet script successfully sent to node bdanode13
INFO: Puppet script successfully sent to node bdanode14
INFO: Puppet script successfully sent to node bdanode15
ERROR: Script could not be sent to some nodes Cannot continue with install #Step 3#
INFO: Time spent in step 3 StopClouderaManager is 330 seconds.
INFO: Running bdadiagcluster...
INFO: Please get the Big Data Appliance cluster diagnostic bundle at /tmp/bdadiagcluster_1465407365.zip
Exiting...

  

Then again in Step 5 a similar error is raised:

ERROR: Sending puppet script to node bda9node05 returned error 3
Warning: Setting manifestdir is deprecated. See http://links.puppetlabs.com/env-settings-deprecations
(at /usr/lib/ruby/site_ruby/1.8/puppet/settings.rb:1095:in 'issue_deprecations')
Warning: Setting modulepath is deprecated in puppet.conf. See http://links.puppetlabs.com/env-settings-deprecations
(at /usr/lib/ruby/site_ruby/1.8/puppet/settings.rb:1095:in 'issue_deprecations')
Warning: Puppet kick is deprecated. See http://links.puppetlabs.com/puppet-kick-deprecation
Warning: Failed to load ruby LDAP library. LDAP functionality will not be available
Triggering bdanode05.example.com
Error: Host bdanode05.example.com failed: No route to host - connect(2)

bdanode05.example.com finished with exit code 2
Failed: bdanode05.example.comINFO: Puppet script successfully sent to node bdanode06
INFO: Puppet script successfully sent to node bdanode07
INFO: Puppet script successfully sent to node bdanode08
INFO: Puppet script successfully sent to node bdanode09
INFO: Puppet script successfully sent to node bdanode10
INFO: Puppet script successfully sent to node bdanode11
INFO: Puppet script successfully sent to node bdanode12
INFO: Puppet script successfully sent to node bdanode13
INFO: Puppet script successfully sent to node bdanode14
INFO: Puppet script successfully sent to node bdanode15
ERROR: Script could not be sent to some nodes Cannot continue with install #Step 5#
INFO: Time spent in step 5 InstallHadoop is 1929 seconds.
INFO: Running bdadiagcluster...
INFO: Please run "setup-root-ssh -C" to setup passwordless SSH to all cluster hosts
Exiting...

In some cases the host or hosts were reachable but most cases the node or nodes become unreachable and had to be started in ILOM via a powercycle.

The nodes in the cluster were running two different versions of the Kernel. In this case the cluster was recently expanded. The original nodes were running a higher kernel than the new nodes recently added via the expansion.

The kernel was also higher on the original node than what the upgrade included.

Additionally a reboot was not asked for on the original nodes with a higher kernel. Only the nodes which were updated with the upgrade were rebooted.

The puppet script was showing a lock file which had to be moved.

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms