[FLINK-26560] Make the threshold of the overlap fraction of incremental restoring configurable

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.15.0
• Fix Version/s: 1.16.0
• Component/s: Runtime / State Backends
• Labels: pull-request-available
• Assignee: Yanfei Lei
• Reporter: Yanfei Lei
• Created: 10/Mar/22 02:46
• Updated: 14/Jun/22 06:17
• Resolved: 18/Mar/22 10:24

Description

Currently, the threshold of the overlap fraction of incremental restoring `OVERLAP_FRACTION_THRESHOLD` is a hard-coded, fixed value.

 

public class RocksDBIncrementalCheckpointUtils {

    /**
     * The threshold of the overlap fraction of the handle's key-group range with target key-group
     * range to be an initial handle.
     */
    private static final double OVERLAP_FRACTION_THRESHOLD = 0.75;
...
} 

 

`OVERLAP_FRACTION_THRESHOLD` is used to control how to restore a state handle, different thresholds can affect the performance of restoring. The behavior of deletion in restoring has been changed after FLINK-21321, the old threshold no longer fits the current situation.

To make it easier to modify the threshold according to different situations, changing `OVERLAP_FRACTION_THRESHOLD` to be configurable is suggested.

Issue Links

• is related to: FLINK-23399 Add a performance benchmark for statebackend rescaling
• is related to: FLINK-21321 Change RocksDB incremental checkpoint re-scaling to use deleteRange
• links to: GitHub Pull Request #19106