BDA Mammoth Upgrade Fails on "Step 8: Cleaning up install" with "ERROR: Ran /usr/bin/scp -r"

(Doc ID 2213619.1)

Last updated on DECEMBER 13, 2016

Applies to:

Big Data Appliance Integrated Software - Version 4.6.0 and later
Linux x86-64

Symptoms

BDA Mammoth upgrade fails at Step 8 "Cleaning up install" with an error like:

INFO: Step 8: Cleaning up install
INFO: Executing updateih.sh on nodes /opt/oracle/BDAMammoth/bdaconfig/tmp/all_nodes #Step 8#
SUCCESS: Executed updateih.sh on nodes /opt/oracle/BDAMammoth/bdaconfig/tmp/all_nodes #Step 8#
SUCCESS: Updated imagehistory on all nodes
INFO: Adding users to groups
INFO: Executing addmember.sh on nodes /opt/oracle/BDAMammoth/bdaconfig/tmp/all_nodes #Step 8#
SUCCESS: Executed addmember.sh on nodes /opt/oracle/BDAMammoth/bdaconfig/tmp/all_nodes #Step 8#
SUCCESS: Succesfully added members to groups
INFO: Saving all install logs
INFO: Making sure all puppet agents can be accessed.
INFO: Pinging puppet agents
INFO: Enable CM email Alerts. This will take some time ...
INFO: Puppet script successfully sent to node bdanode01
INFO: Pinging puppet agents ...................
INFO: Puppet agent run on node bdanode01 completed successfully.
INFO: Check the log file in /opt/oracle/BDAMammoth/bdaconfig/tmp/pagent-bdanode01-.log
INFO: Creating environment.pp file ...
INFO: Making sure all puppet agents can be accessed.
INFO: Pinging puppet agents .
INFO: Cleaning up install. This will take some time ...
INFO: Puppet script successfully sent to node bdanode01
...
INFO: Puppet script successfully sent to node bdanode0n

INFO: Pinging puppet agents ..
INFO: Puppet agent run on node bdanode01 completed successfully.
INFO: Check the log file in /opt/oracle/BDAMammoth/bdaconfig/tmp/pagent-bdanode01-.log
INFO: Puppet agent run on node bdanode02 completed successfully.
INFO: Check the log file in /opt/oracle/BDAMammoth/bdaconfig/tmp/pagent-bdanode02-.log
...
INFO: Puppet agent run on node bdanode0n completed successfully.
INFO: Check the log file in /opt/oracle/BDAMammoth/bdaconfig/tmp/pagent-bdanode0n-.log

SUCCESS: Ran /usr/bin/scp -r bdanode02:/opt/oracle/BDAMammoth/bdaconfig/tmp/* /opt/oracle/bda/install/log/upgrade-/tmp-bdanode02 and it returned: RC=0
SUCCESS: Ran /bin/rm -rf /opt/oracle/bda/install/log/upgrade-/tmp-bdanode02/*.sh /opt/oracle/bda/install/log/upgrade-/tmp-bdanode02/*.pp /opt/oracle/bda/install/log/upgrade-/tmp-bdanode02/*.pl and it returned: RC=0
SUCCESS: Ran /usr/bin/ssh bdanode02 "/bin/rm -rf /opt/oracle/BDAMammoth/bdaconfig/tmp/*" and it returned: RC=0

SUCCESS: Ran /usr/bin/scp -r bdanode03:/opt/oracle/BDAMammoth/bdaconfig/tmp/* /opt/oracle/bda/install/log/upgrade-/tmp-bdanode03 and it returned: RC=0
SUCCESS: Ran /bin/rm -rf /opt/oracle/bda/install/log/upgrade-/tmp-bdanode03/*.sh /opt/oracle/bda/install/log/upgrade-/tmp-bdanode03/*.pp /opt/oracle/bda/install/log/upgrade-/tmp-bdanode03/*.pl and it returned: RC=0
...
SUCCESS: Ran /usr/bin/scp -r bdanode14:/opt/oracle/BDAMammoth/bdaconfig/tmp/* /opt/oracle/bda/install/log/upgrade-/tmp-bdanode14 and it returned: RC=0
SUCCESS: Ran /bin/rm -rf /opt/oracle/bda/install/log/upgrade-/tmp-bdanode14/*.sh /opt/oracle/bda/install/log/upgrade-/tmp-bdanode14/*.pp /opt/oracle/bda/install/log/upgrade-/tmp-bdanode14/*.pl and it returned: RC=0
SUCCESS: Ran /usr/bin/ssh bdanode14 "/bin/rm -rf /opt/oracle/BDAMammoth/bdaconfig/tmp/*" and it returned: RC=0

ERROR: Ran /usr/bin/scp -r bdanode15:/opt/oracle/BDAMammoth/bdaconfig/tmp/* /opt/oracle/bda/install/log/upgrade-/tmp-bdanode15 and it returned code(1) instead of 0 and output is
scp: /opt/oracle/BDAMammoth/bdaconfig/tmp/call_cm.err: No such file or directory
scp: /opt/oracle/BDAMammoth/bdaconfig/tmp/call_cm.out: No such file or directory

INFO: Time spent in step 8 CleanupInstall is 3232 seconds.
INFO: Running bdadiagcluster...
INFO: Please get the Big Data Appliance cluster diagnostic bundle at /tmp/bdadiagcluster_.zip
Exiting...


Using an example of upgrade from BDA V4.4 to BDA V4.6 other symptoms are like:

1.  "imageinfo" is not updated. It reports the pre-upgrade versions. For example on upgrade from BDA V4.4 to BDA V4.6, imageinfo reports 4.4 values:


5.  Trying to run the cluster verifications with "./mammoth -c" at this point will fail because "imageinfo" and the "config.json" are not yet updated.

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms