Slave Stops With Error_code: 1296; handler error HA_ERR_ROWS_EVENT_APPLY

(Doc ID 2271733.1)

Last updated on AUGUST 25, 2017

Applies to:

MySQL Cluster - Version 7.3 to 7.6 [Release 7.3 to 7.0]
Information in this document applies to any platform.

Symptoms

Setup and test performed:

  1. Two clusters each one has two mysqld processes.
  2. Replication setup of two possible replication channels.
  3. Replication starts with channel one up and running.
  4. On one of the slaves we issue: stop slave.
  5. The failover job is initiated to switch to channel two - second set of mysqld process master and slave.
  6. Switch over succeeds on that cluster and replication may continue.
  7. The other cluster slave fails with the error we show.

To switch over implement the following: https://dev.mysql.com/doc/mysql-cluster-excerpt/5.1/en/mysql-cluster-replication-failover.html

mysqlS'> SELECT @latest:=MAX(epoch) FROM mysql.ndb_apply_status;
mysqlM'> SELECT @file:=SUBSTRING_INDEX(File, '/', -1), @pos:=Position FROM mysql.ndb_binlog_index WHERE epoch > @latest ORDER BY epoch ASC LIMIT 1;

Log:

2015-04-28 11:57:13 32127 [Note] NDB Slave : MaxReplicatedEpoch set to 0
(0/0) at Slave start
2015-04-28 12:17:03 32127 [Warning] NDB: Thread id 1 timed out (30s) waiting
for epoch 36514/9 to be handled. Progress : 36514/8 -> 36514/8.
2015-04-28 12:17:33 32127 [Warning] NDB: Thread id 1 timed out (30s) waiting
for epoch 36529/1 to be handled. Progress : 36514/8 -> 36514/8.
2015-04-28 12:33:57 32127 [Warning] NDB: Thread id 1 timed out (30s) waiting
for epoch 37004/19 to be handled. Progress : 37004/17 -> 37004/17.
.......................
2015-04-28 16:30:00 32127 [Warning] NDB: Thread id 1 timed out (30s) waiting
for epoch 43845/8 to be handled. Progress : 43845/7 -> 43845/7.
2015-04-28 16:30:19 32127 [Warning] NDB: Thread id 607 timed out (30s)
waiting for epoch 43854/13 to be handled. Progress : 43845/7 -> 43845/7.
2015-04-28 16:30:30 32127 [Warning] NDB: Thread id 1 timed out (30s) waiting
for epoch 43854/13 to be handled. Progress : 43845/7 -> 43845/7.
.......................
2015-04-29 14:58:13 32127 [Warning] NDB: Thread id 1 timed out (30s) waiting
for epoch 82899/9 to be handled. Progress : 82899/8 -> 82899/8.
2015-04-29 14:58:43 32127 [Warning] NDB: Thread id 1 timed out (30s) waiting
for epoch 82914/0 to be handled. Progress : 82899/8 -> 82899/8.
2015-04-29 15:06:32 32127 [Note] Error reading relay log event: slave SQL
thread was killed
2015-04-29 15:06:32 32127 [Note] Slave I/O thread exiting, read up to log
'mysqld_52_WEST_bin.000089', position 470487624
2015-04-29 15:08:38 32127 [Note] 'CHANGE MASTER TO executed'. Previous state
master_host='', master_port= 3306, master_log_file='', master_log_pos= 4,
master_bind=''. New state master_host='XX.XX.XX.XXX', master_port= 3306,
master_log_file='mysqld_52_WEST_bin.000089', master_log_pos= 556371456,
master_bind=''.

2015-04-29 15:08:38 32127 [Warning] Storing MySQL user name or password
information in the master info repository is not secure and is therefore not
recommended. Please consider using the USER and PASSWORD connection options
for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more
information.
2015-04-29 15:08:38 32127 [Warning] Slave SQL: If a crash happens this
configuration does not guarantee that the relay log info will be consistent,
Error_code: 0
2015-04-29 15:08:38 32127 [Note] Slave SQL thread initialized, starting
replication in log 'mysqld_52_WEST_bin.000089' at position 556371456, relay
log './mdsuapp1a-relay-bin.000001' position: 4
2015-04-29 15:08:38 32127 [Note] Slave I/O thread: connected to master
'rep@22.22.20.109:3306',replication started in log
'mysqld_52_WEST_bin.000089' at position 556371456
2015-04-29 15:08:38 32127 [Note] NDB Slave : MaxReplicatedEpoch set to 0
(0/0) at Slave start
2015-04-29 15:10:30 32127 [Warning] NDB Slave : ERROR. SQL thread stopped as
applying epoch 83778/2 (359823770124290) from Master ServerId 208 which is
lower than previously applied epoch 83836/16 (360072878227472). Group Master
Log : mysqld_52_WEST_bin.000089 Group Master Log Pos : 643980719.
2015-04-29 15:10:30 32127 [ERROR] Slave SQL: Could not execute Write_rows
event on table mysql.ndb_apply_status; Got error 173 'Unknown error code'
from NDBCLUSTER, Error_code: 1296; handler error HA_ERR_ROWS_EVENT_APPLY; the
event's master log mysqld_52_WEST_bin.000089, end_log_pos 643985881,
Error_code: 1296
2015-04-29 15:10:30 32127 [Warning] Slave: Got error 173 'Unknown error code'
from NDBCLUSTER Error_code: 1296
2015-04-29 15:10:30 32127 [ERROR] Error running query, slave SQL thread
aborted. Fix the problem, and restart the slave SQL thread with "SLAVE
START". We stopped at log 'mysqld_52_WEST_bin.000089' position 643980719
2015-04-29 15:16:04 32127 [ERROR] Error reading packet from server: Lost
connection to MySQL server during query ( server_errno=2013)
2015-04-29 15:16:04 32127 [Note] Slave I/O thread killed while reading event
2015-04-29 15:16:04 32127 [Note] Slave I/O thread exiting, read up to log
'mysqld_52_WEST_bin.000089', position 888434806
2015-04-30 11:08:20 32127 [Note] /usr/local/mysql-cluster_x86_64/bin/mysqld:
Normal shutdown

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms