[FLINK-21321] Change RocksDB incremental checkpoint re-scaling to use deleteRange

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.16.0
• Component/s: Runtime / State Backends
• Labels: pull-request-available
• Assignee: yanfei lei
• Reporter: Joey Pereira
• Created: 08/Feb/21 13:49
• Updated: 14/Jun/22 06:22
• Resolved: 21/Mar/22 03:14

Description

In FLINK-8790, it was suggested to use RocksDB's deleteRange API to more efficiently clip the databases for the desired target group.

During the PR for that ticket, #5582, the change did not end up using the deleteRange method as it was an experimental feature in RocksDB.

At this point deleteRange is in a far less experimental state now but I believe is still formally "experimental". It is heavily by many others like CockroachDB and TiKV and they have teased out several bugs in complex interactions over the years.

For certain re-scaling situations where restores trigger restoreWithScaling and the DB clipping logic, this would likely reduce an O[n] operation (N = state size/records) to O(1). For large state apps, this would potentially represent a non-trivial amount of time spent for re-scaling. In the case of my workplace, we have an operator with 100s of billions of records in state and re-scaling was taking a long time (>>30min, but it has been awhile since doing it).

Issue Links

• causes: FLINK-26789 RescaleCheckpointManuallyITCase.testCheckpointRescalingInKeyedState failed
• is a child of: FLINK-28038 RocksDB rescaling improvement & rescaling benchmark - umbrella ticket
• is blocked by: FLINK-14482 Bump up rocksdb version
• relates to: FLINK-26560 Make the threshold of the overlap fraction of incremental restoring configurable
• relates to: FLINK-23399 Add a performance benchmark for statebackend rescaling
• relates to: FLINK-8790 Improve performance for recovery from incremental checkpoint
• links to: GitHub Pull Request #14893
• links to: GitHub Pull Request #19033