MySQL Cluster Does Not Restart Due to Ndbd File System Inconsistency Error During Startphase 4 (Doc ID 1364988.1)

Last updated on MARCH 22, 2016

Applies to:

MySQL Cluster - Version 7.1 and later
Information in this document applies to any platform.

Symptoms

MySQL Cluster does not restart due to Ndbd file system inconsistency errors on a data node.
The entire MySQL Cluster is down.

In the Cluster Error Log we can verify that the problem occurs during the Start Phase 4 of the corrupted node (in this example Node 4):

[MgmSrvr] INFO -- Node 4: Node 3: API mysql-5.1.51 ndb-6.3.39
[MgmSrvr] INFO -- Node 4: Start phase 1 completed
[MgmSrvr] INFO -- Node 3: Start phase 2 completed (system restart)
[MgmSrvr] INFO -- Node 4: Start phase 2 completed (system restart)
[MgmSrvr] INFO -- Node 3: Start phase 3 completed (system restart)
[MgmSrvr] INFO -- Node 4: Start phase 3 completed (system restart)

[MgmSrvr] ALERT -- Node 1: Node 4 Disconnected

[MgmSrvr] ALERT -- Node 4: Forced node shutdown completed. Occurred during startphase 4. Caused by error 2815: 'File not found(Ndbd file system inconsistency error, please report a bug). Ndbd file system error, restart node initial'.

 

An alternative error on Node 4 that will cause the same problem at cluster level is Error 2352:

[MgmSrvr] ALERT -- Node 4: Forced node shutdown completed. Occurred during startphase 4. Caused by error 2352: 'Invalid LCP(Ndbd file system inconsistency error, please report a bug). Ndbd file system error, restart node initial'.

 

After Node 4 is forced to shutdown (due to errors 2815 or 2352), the other nodes are forced to shutdown due to error 2308:

2011-10-05 15:10:40 [MgmSrvr] ALERT -- Node 1: Node 3 Disconnected

2011-10-05 15:10:40 [MgmSrvr] ALERT -- Node 3: Forced node shutdown completed. Occured during startphase 4. Caused by error 2308: 'Another node failed during system restart, please investigate error(s) on other node(s)(Restart error). Temporary error, restart node'.

 

As a result the entire cluster does not start.

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms