[FLINK-21694] Increase default value of "state.backend.rocksdb.checkpoint.transfer.thread.num"

• Type: Improvement
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 1.13.0
• Component/s: Runtime / State Backends
• Assignee: Stephan Ewen
• Reporter: Stephan Ewen
• Created: 09/Mar/21 20:27
• Updated: 20/Apr/21 20:11
• Resolved: 20/Apr/21 16:14

Description

The default value for the number of threads used to download state artifacts from checkpoint storage should be increased.

The increase should not pose risk of regression, but does in many cases speed up checkpoint recovery significantly.

Something similar was reported in this blog post, item (3).
https://engineering.contentsquare.com/2021/ten-flink-gotchas/

A default value of 8 (eight) sounds like a good default. It should not result in excessive thread explosion, and already speeds up recovery.

Issue Links

• causes: FLINK-22381 RocksDBStateBackendTests::test_get_set_number_of_transfering_threads fails on azure