My Oracle Support Banner

MySQL Innodb Cluster Sometimes Does Not Automatically Recover After Restarting One Node. (Doc ID 2927786.1)

Last updated on FEBRUARY 13, 2023

Applies to:

MySQL Server - Version 8.0 and later
Information in this document applies to any platform.

Symptoms

On : 8.0 version, Group Replication

ACTUAL BEHAVIOR
---------------
MySQL innodb cluster sometimes does not automatically recover after restarting one node.

Performing a one node failure test to check the resiliency of the db cluster, but the cluster does not return to normal under a certain condition.

Environment
- innodb cluster managed by MySQL Operator 8.0.28 with 3 instances and 1 router running on Red Hat OpenShift 4.9.19 (Kubernetes).  
- The MySQL pods are controlled by a statefulset and one pod running on each worker node. 

What happened:
1. Terminated one node (worker-0), the pod (mycluster-2) went down with pending state. Other pods were running normally.
2. restarted the node (worker-0) and the pod (mycluster-2) came back to normal state. 
3. mycluster-2 appeared to recover automatically in pod status, but I got a waring message below when I checked the cluster status with dba.getCluster().status() by mysqlsh.

"WARNING: Instance is not managed by InnoDB cluster. Use cluster.rescan() to repair."

Connection to mycluster-2 via mysqlsh was not possible.




EXPECTED BEHAVIOR
-----------------------
When the pod comes back online in normal state, it should be automatic and not require a manual cluster.rescan() command.

STEPS
-----------------------
The issue can be reproduced at will with the following steps:
1. Have a setup using Kubernetes using assigned worker nodes and pods for mysql innodb cluster. Terminate one node (worker-0).
2. Restart the node (worker-0) to bring pod back online.
3. Cluster pod appears to be online, but warning appears and requires a cluster.rescan() command.

BUSINESS IMPACT
-----------------------
The issue has the following business impact:
Due to this issue, users cannot get a fully automated implementation of InnoDB Cluster .

Changes

 

Cause

To view full details, sign in with your My Oracle Support account.

Don't have a My Oracle Support account? Click to get started!


In this Document
Symptoms
Changes
Cause
Solution


My Oracle Support provides customers with access to over a million knowledge articles and a vibrant support community of peers and Oracle experts.