[IGNITE-17793] Historical rebalance must use HWM instead of LWM to seek the proper checkpoint to avoid the data loss

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.15
• Labels: iep-31 ise
• Release Note: Fixed potential data loss on historical rebalance
• Ignite Flags: Docs Required, Release Notes Required
• Assignee: Vladimir Steshin
• Reporter: Anton Vinogradov
• Created: 30/Sep/22 17:43
• Updated: 24/Apr/23 15:11
• Resolved: 07/Dec/22 13:44

Description

Currently, historical rebalance at CheckpointHistory#searchEarliestWalPointer seeks for the newest checkpoint with counter less that lowest entry has to be rebalanced.

Unfortunately, we may have more that one checkpoint with the same counter and it's impossible to use the newest one as a rebalance start point.

For example, we have partition with LWM=100, some gaps and HWM=200.
Checkpoint will have the counter == 100.
Then we may close some gaps, exluding 101 (to keep LWM == 100).
And again, checkpoint will have counter == 100.
Newest checkpoint (marked with counter 100) will not cointain all committed entries with counter > 100.
Then lets close the rest of the gaps to make historical rebalance possible.
And after the rebalance finish, we'll see a warning "Some partition entries were missed during historical rebalance" and inconsistent cluster state.

See reproducer at HistoricalRebalanceCheckpointTest.java

Possible solution is to use HWM instead of LWM during the search.

Issue Links

• Dependency: IGNITE-17908 AssertionError LWM after reserved on data insertion after the cluster restart
• is related to: IGNITE-18343 Refactor partition counters API.
• links to: GitHub PullRequest #10396, v2: store HWM to WAL