[FLINK-32345] Improve parallel download of RocksDB incremental state

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.18.0
• Component/s: Runtime / State Backends
• Labels: pull-request-available
• Assignee: Stefan Richter
• Reporter: Stefan Richter
• Created: 15/Jun/23 08:25
• Updated: 28/Jul/23 13:09
• Resolved: 27/Jun/23 13:19

Description

RocksDBStateDownloader is used to download the files for incremental checkpoints in parallel. However, the parallelism is currently restricted to a single IncrementalRemoteKeyedStateHandle and also a single state type (shared, private) within the handle at a time.
We should support parallelization across multiple state types and across multiple state handles. In particular, this can improve our download times for scale-in.

Issue Links

• causes: FLINK-32681 RocksDBStateDownloaderTest.testMultiThreadCleanupOnFailure unstablie
• relates to: FLINK-10461 Speed up RocksDB download file procedure when restoring
• links to: GitHub Pull Request #22788
• mentioned in: Page Loading...