Node 1 of the Cluster Goes Down During Step 1 of Mammoth Upgrade During "Patching factory image" and After Reboot is Found to Have an Upgraded Kernel and JDK

(Doc ID 2317069.1)

Last updated on NOVEMBER 22, 2017

Applies to:

Big Data Appliance Integrated Software - Version 4.9.0 and later
Linux x86-64

Symptoms

During Step 1 of a Mammoth upgrade when patching the factory image at the step "Patching factory image" Node 1 is unexpectedly found to be down.

1. The Mammoth Step 1 output at this point looks like:

...

INFO: Patching factory image. This will take some time ...
INFO: Puppet script successfully sent to node bdanode01
INFO: Pinging puppet agents .............................................................................................
INFO: Puppet agent run on node bdanode01 completed successfully.
INFO: Check the log file in /opt/oracle/BDAMammoth/bdaconfig/tmp/pagent-bdanode01-<timestamp>.log
INFO: Making sure all puppet agents can be accessed.
INFO: Pinging puppet agents
INFO: Patching factory image. This will take some time ...
ERROR: Sending puppet script to node bdanode02 returned error 3
ERROR: Sending puppet script to node bdanode03 returned error 3
ERROR: Sending puppet script to node bdanode04 returned error 3
...

 

2. Node 1 of the cluster is found to be down and requires a restart via an ILOM power cycle, using: How to Power Cycle an Oracle Big Data Appliance Node Using the ILOM when the Node is Not Reachable Using Ping/SSH (Doc ID 1550440.1).

3. Once Node 1 is back up the following symptoms are observed:

a)  Running "dcli -C uname -a" shows that the kernel is upgraded on Node 1 but not on the other nodes.

In the case of an upgrade from BDA V4.7 to BDA V4.9 output looks like below showing that the kernel is only upgraded on Node 1: 

c) Cloudera Manager shows no heartbeat into Node 1

 

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms