Starting up a failed data node causes UP A FAILED DATA NODE CAUSES "A FOREIGN KEY CONSTRAINT FAILS" (Doc ID 1912930.1)

Last updated on FEBRUARY 16, 2016

Applies to:

MySQL Cluster - Version 7.3 and later
Information in this document applies to any platform.

Symptoms

Error log:

com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Cannot add or update a child row: a foreign key constraint fails (Unknown error code)

Sequence of events:

  1. Node 4 shuts down
  2. Few minutes later the Node 4 is manually started back up
  3. As soon as Node 4 starts the FK problems appear
  4. So stopped Node 4 and confirm the problem is no longer present when Node 4 is shut down
  5. Tried starting Node 4 and this causes the problem to start occurring again
  6. Restart Node 4 with --initial hoping that this will resolve the problem
  7. After the --initial startup on Node 4 is complete, the problem starts occurring again
  8. Shut Node 4 down again as this is seriously affecting a live website
  9. Currently running our SQL cluster on a single data node (Node 3) because of this issue

Have not restarted Node 3 as it doesn't seem to be causing any issues and haven't restarted the SQL nodes either as both of them work correctly when data Node 4 is shut down.

Node 3 Error Log

Time: Sunday 27 April 2014 - 18:56:36
Status:  Temporary error, restart node
Message: Node lost connection to other nodes and can not form a unpartitioned
cluster, please investigate if there are error(s) on other node(s)
(Arbitration error)
Error: 2305
Error data: Arbitrator decided to shutdown this node
Error object: QMGR (Line: 6068) 0x00000002
Program: ndbmtd
Pid: 18466 thr: 0
Version: mysql-5.6.15 ndb-7.3.4
Trace: /data/mysql-cluster/ndbmtd/logs/ndb_3_trace.log.4 [t1..t7]
***EOM***
Time: Friday 2 May 2014 - 13:38:36
Status: Temporary error, restart node
Message: Node lost connection to other nodes and can not form a unpartitioned
cluster, please investigate if there are error(s) on other node(s)
(Arbitration error)
Error: 2305
Error data: Arbitrator decided to shutdown this node
Error object: QMGR (Line: 6068) 0x00000002
Program: ndbmtd
Pid: 6035 thr: 0
Version: mysql-5.6.15 ndb-7.3.4
Trace: /data/mysql-cluster/ndbmtd/logs/ndb_3_trace.log.5 [t1..t7]
***EOM***


Node 4 error log

Time: Monday 28 April 2014 - 09:22:25
Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error,
programming error or missing error message, please report a bug)
Error: 2341
Error data: DbtcMain.cpp
Error object: DBTC (Line: 18290) 0x00000002
Program: ndbmtd
Pid: 2031 thr: 0
Version: mysql-5.6.15 ndb-7.3.4
Trace: /data/mysql-cluster/ndbmtd/logs/ndb_4_trace.log.10 [t1..t7]
***EOM***
Time: Friday 2 May 2014 - 13:57:58
Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error,
programming error or missing error message, please report a bug)
Error: 2341
Error data: DbtcMain.cpp
Error object: DBTC (Line: 18290) 0x00000002
Program: ndbmtd
Pid: 18583 thr: 0
Version: mysql-5.6.15 ndb-7.3.4
Trace: /data/mysql-cluster/ndbmtd/logs/ndb_4_trace.log.11 [t1..t7]
***EOM***
Time: Thursday 8 May 2014 - 20:45:10
Status: Temporary error, restart node
Message: Node lost connection to other nodes and can not form a unpartitioned
cluster, please investigate if there are error(s) on other node(s)
(Arbitration error)
Error: 2305
Error data: Arbitrator decided to shutdown this node
Error object: QMGR (Line: 6068) 0x00000002
Program: ndbmtd
Pid: 28013 thr: 0
Version: mysql-5.6.15 ndb-7.3.4
Trace: /data/mysql-cluster/ndbmtd/logs/ndb_4_trace.log.12 [t1..t7]
***EOM***

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms