NDB Continuously Rebooting Due To File Not Found (DBLQH: Caused By Error 2341) (Doc ID 2095347.1)

Last updated on JANUARY 08, 2016

Applies to:

MySQL Cluster - Version 7.1 to 7.4 [Release 7.1 to 7.4]
Information in this document applies to any platform.

Symptoms

When attempting to restart a Data node one of the following errors occur:

2015-05-20 01:45:24 [MgmtSrvr] WARNING -- Node 3: Node 4 missed heartbeat 2
2015-05-20 01:45:25 [MgmtSrvr] WARNING -- Node 3: Node 4 missed heartbeat 3
2015-05-20 01:45:28 [MgmtSrvr] ALERT -- Node 1: Node 4 Disconnected
2015-05-20 01:45:28 [MgmtSrvr] ALERT -- Node 4: Forced node shutdown completed. Occured during startphase 4. Caused by error 2341: 'Internal program error (failed ndbrequire)(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.
2015-12-14 09:38:46 [MgmtSrvr] INFO -- Node 4: Node restart starting to copy the fragments to Node 4
...
2015-12-14 09:42:25 [MgmtSrvr] WARNING -- Node 3: Node 4 missed heartbeat 2
2015-12-14 09:42:26 [MgmtSrvr] WARNING -- Node 3: Node 4 missed heartbeat 3
2015-12-14 09:42:27 [MgmtSrvr] WARNING -- Node 3: Node 4 missed heartbeat 4
2015-12-14 09:42:27 [MgmtSrvr] ALERT -- Node 3: Node 4 declared dead due to missed heartbeat
2015-12-14 09:42:27 [MgmtSrvr] INFO -- Node 3: Communication to Node 4 closed
2015-12-14 09:42:27 [MgmtSrvr] ALERT -- Node 3: Network partitioning - arbitration required
2015-12-14 09:42:27 [MgmtSrvr] INFO -- Node 3: President restarts arbitration thread [state=7]
2015-12-14 09:42:27 [MgmtSrvr] ALERT -- Node 1: Node 4 Disconnected
2015-12-14 09:42:27 [MgmtSrvr] ALERT -- Node 3: Arbitration won - positive reply from node 2
2015-12-14 09:42:27 [MgmtSrvr] ALERT -- Node 3: Node 4 Disconnected
2015-12-14 09:42:28 [MgmtSrvr] INFO -- Node 3: Started arbitrator node 1 [ticket=6be9000239f7eadc]
2015-12-14 09:42:29 [MgmtSrvr] ALERT -- Node 4: Forced node shutdown completed. Occured during startphase 4. Caused by error 2341: 'Internal program error (failed ndbrequire)(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.

ndb_4_error.log:

Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error, programming error or missing error message, please report a bug)
Error: 2341
Error data: DblqhMain.cpp
Error object: DBLQH (Line: 17431) 0x00000002

Trace log:

RESTORE 000387
DBLQH 003487
DBTUP 010031
DBLQH 017430 017431

--------------- Signal ----------------
r.bn: 247/1 "DBLQH", r.proc: 4, r.sigId: 32339591 gsn: 90 "RESTORE_LCP_REF" prio: 1
s.bn: 262/1 "RESTORE", s.proc: 4, s.sigId: 32339590 length: 4 trace: 0 #sec: 0 fragInf: 0
H'0000022f H'00000aff H'00000aff H'00000002
--------------- Signal ----------------
r.bn: 262/1 "RESTORE", r.proc: 4, r.sigId: 32339590 gsn: 260 "FSOPENREF" prio: 1
s.bn: 253 "NDBFS", s.proc: 4, s.sigId: 585024 length: 4 trace: 0 #sec: 0 fragInf: 0
UserPointer: 0
ErrorCode: 2815, File not found
OS ErrorCode: 2
--------------- Signal ----------------
r.bn: 262/1 "RESTORE", r.proc: 4, r.sigId: 32339589 gsn: 91 "RESTORE_LCP_REQ" prio: 1
s.bn: 247/1 "DBLQH", s.proc: 4, s.sigId: 32339588 length: 6 trace: 0 #sec: 0 fragInf: 0
H'0000022f H'02f70004 H'00000000 H'000001a7 H'00000005 H'00005a8a
--------------- Signal ----------------
r.bn: 247/1 "DBLQH", r.proc: 4, r.sigId: 32339588 gsn: 89 "RESTORE_LCP_CONF" prio: 1
s.bn: 249/1 "DBTUP", s.proc: 4, s.sigId: 32339587 length: 2 trace: 0 #sec: 0 fragInf: 0
H'0000022e H'02f90004

Cause

Sign In with your My Oracle Support account

Don't have a My Oracle Support account? Click to get started

My Oracle Support provides customers with access to over a
Million Knowledge Articles and hundreds of Community platforms