[FLINK-10461] Speed up RocksDB download file procedure when restoring

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.8.0
• Component/s: Runtime / State Backends
• Labels: pull-request-available
• Assignee: Congxian Qiu
• Reporter: Congxian Qiu
• Created: 29/Sep/18 02:29
• Updated: 15/Jun/23 08:26
• Resolved: 20/Dec/18 10:06

Description

In the current master branch, the restore will download file from DFS, the download procedure are single-thread, this could speed up by using multi-thread for downloading states from DFS.

 

In my company, the states will come to some terabytes, so the restore procedure will become a litter slow, after a bit digging, I find download states from DFS using single thread, this could using multi-thread for speed up.

I test the time used for download states from DFS with ~2 terabytes states. With single thread it used 640+s, and 130+s when using 5 threads for download.

Issue Links

• is related to: FLINK-32345 Improve parallel download of RocksDB incremental state
• links to: GitHub Pull Request #6777