Temporary Disk I/O Stalls; LCP Frag Watchdog : No progress on Table
(Doc ID 1582813.1)
Last updated on DECEMBER 18, 2017
Applies to:MySQL Cluster - Version 6.3 and later
During periods with high load, for example while creating backups or executing bulk updates particularly if these involve large BLOBs, disk I/O activities are temporary stalling.
For performance critical I/O activities this may cause temporary errors and/or the data node to crash. Examples of performance critical I/O in MySQL Cluster are:
- Local Checkpoints (LCPs). If no progress is detected for a period of LcpScanProgressTimeout (defaults to 60 seconds; in 7.2 before 7.2.14 and in 7.3 before 7.3.3 this was hardcoded to 60 seconds), the LCP Frag watchdog shuts down the data node.
- REDO logs. If the REDO logs takes too long to flush, pending transactions will be aborted.
One sign of the issue - although not necessarily required - is 100% disk utilisation while nothing is written to or read from the disk, for example:
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 5.01 0.00 0.00 100.20
The more memory that is available on the server, the more likely this issue is to happen, so the issue may begin after installing more memory or upgrading to a new server with more memory available.
To view full details, sign in with your My Oracle Support account.
Don't have a My Oracle Support account? Click to get started!
In this Document