Affects Version/s: None
Fix Version/s: None
Component/s: Database Core
When a db compaction is interrupted and restarted, it's possible for the most recent header written to the .compact.meta file to be buried under many GB from the end of file. For example, an interrupted compaction left the following files:
which means the current algorithm must search through about 8GB of file before it gets to a header. I measured how long it takes, both on an SSD laptop and SSD server, getting similar numbers:
which is between 328-427 seconds, or 19-25 MB/s.
One reason for this relative slowness is because the current algorithm performs a disk read for every block it searches:
We can improve the speed by loading a "chunk" of many blocks into memory with a single read operation, and then search each block in memory, trading off memory for speed. Ideally, the tradeoff can be made configurable, so that existing speed/memory behavior can be retained by default.