How to Migrate the MySQL Master from the MySQL Master Host if the MySQL Master Host Goes Down on BDA 4.5 and Higher (Doc ID 2611637.1)

Last updated on APRIL 17, 2023

Applies to:

Big Data Appliance X6-2 Hardware - Version All Versions and later
x86_64

Purpose

Background

In the case of the MySQL Master host, Node 3 by default, going down due to a server failure, OS corruption, etc. the steps here provide a way to migrate the MySQL master to a new host. This provides a faster method for keeping MySQL functional than performing a full MySQL Master host migration as outlined in:

Node 3 Migration and Reprovision on Oracle Big Data Appliance V4.1 OL6 Hadoop Cluster to Manage a Hardware Failure (Doc ID 1984854.1)
Node Migration on Oracle Big Data Appliance V4.0 OL6 Hadoop Cluster to Manage a Hardware Failure (Doc ID 1946210.1)

After migrating the MySQL Master the rest of the node migration steps can be performed at any time. Note that Cloudera Manager(CM) and the rest of the roles on the Node with MySQL Master role, Node 3 by default, will be unavailable until the node is fully migrated. When the MySQL Master is migrated it will be migrated to the first available noncritical node, generally Node 5. When the rest of the node migration is performed the rest of the roles will be migrated to that same node as well. By default you can not select the noncritical node the MySQL master or other services from that host will be migrated to.

These steps to migrate the MySQL Master are supported on Mammoth 4.5 and higher and require a cluster of 5 or more nodes.

On BDA 4.5 through BDA 5.1 a patch is required to migrate the MySQL Master independently of the rest of the roles. Open an SR with Oracle Support to request the Mammoth version specific patch.

The patched files are below. Note that migratemysqlfiles.erb does not exist on 4.5 to 5.1 clusters. It is obtained via the patch.

/opt/oracle/BDAMammoth/puppet/modules/mysql/templates/changemaster.erb
/opt/oracle/BDAMammoth/puppet/modules/mysql/manifests/importmysql.pp
/opt/oracle/BDAMammoth/puppet/modules/mysql/templates/migratemysqlfiles.erb

For BDA 4.5 there are 3 additional patched files plus additional patching is required for changemaster.erb.

/opt/oracle/BDAMammoth/puppet/modules/mysql/templates/changemaster.erb
/opt/oracle/BDAMammoth/bdaconfig/MammothSteps/InitialChecks.pm
/opt/oracle/BDAMammoth/puppet/modules/hadoop/manifests/startsvc2.pp
/opt/oracle/BDAMammoth/puppet/modules/hadoop/manifests/startscm.pp

Note: For testing the MySQL Master host, Node 3 by default, needs to be shutdown with something like "shutdown -hP now".

Overview of the MySQL Master Migration Steps

1. Replace the Mammoth version specific patch files:

/opt/oracle/BDAMammoth/puppet/modules/mysql/templates/changemaster.erb
/opt/oracle/BDAMammoth/puppet/modules/mysql/manifests/importmysql.pp
/opt/oracle/BDAMammoth/puppet/modules/mysql/templates/migratemysqlfiles.erb

2. Edit the SetupMySQL function in SetupMySQL.pm to stop after MySQL Master migration.

3. Run node migration for the host with MySQL Master role, Node 3 by default.

4. Perform post MySQL Master migration checks

5. Redirect traffic from Node 3 (or the old master) to Node 5 (the new master) on the critical cluster nodes.

6. When ready resume migration after updating StepState.out.

7. When node migration is complete, remove the rules redirecting traffic from Node 3 (or the old master).

Scope

Anyone needing to migrate the MySQL Master in the case of the MySQL Master Host, Node 3 by default, going down prior to performing a full node migration.