[FLINK-8602] Improve recovery performance for rocksdb backend

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Done
• Affects Version/s: 1.5.0
• Component/s: Runtime / State Backends
• Assignee: Sihua Zhou
• Reporter: Sihua Zhou
• Created: 08/Feb/18 08:21
• Updated: 05/Jun/18 13:15
• Resolved: 05/Jun/18 13:15

Description

Right now, RocksDB backend supports fully checkpoint & incremental checkpoint. This issue try to improve the performance of recovery from fully checkpoint as well as from incremental checkpoint. It contains 2 sub-tasks.

1. improve recovery performance for incremental checkpoint when hasExtraKeys = true
2. introduce `parallel recovery` from both fully checkpoint and incremental checkpoint (Base on ingestExternalFile() and SstFileWriter provided by RocksDB).

Any advice would be highly appreciated!

Issue Links

• is a parent of: FLINK-8790 Improve performance for recovery from incremental checkpoint
• is a parent of: FLINK-8845 Use WriteBatch to improve performance for recovery in RocksDB backend
• is a parent of: FLINK-8846 Introduce `parallel recovery` mode for incremental checkpoint