[IGNITE-16657] [Native Persistence 3.0] Implement partitions chunks merger

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0-beta1
• Labels: ignite-3
• Epic Link: New PageMemory Persistent Store
• Assignee: Kirill Tkalenko
• Reporter: Ivan Bessonov
• Reviewer: Semyon Danilov
• Created: 05/Mar/22 08:19
• Updated: 13/Oct/22 12:01
• Resolved: 04/Aug/22 09:15

Description

At the moment, after each checkpoint, delta files are created for each modified partition (which have dirty pages).

We need to implement merging the delta file into the main partition file to reduce read amplification.

Suggested solution:

• Merge to make multi-thread, the number of threads to set in the configuration;
• Each thread processes one delta file at a time;
• Merge to start as soon as there are delta files.

The question remains, what to do when a checkpoint and merge of delta files are working at the same time, who should be given priority and how to deal with it? It is proposed to think over and do in IGNITE-17446.

Issue Links

• Dependent: IGNITE-15818 [Native Persistence 3.0] Checkpoint, lifecycle and file store refactoring and re-implementation
• is a parent of: IGNITE-17446 Determine priority between checkpoint and merge delta files
• links to: GitHub Pull Request #972