Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-21694

Increase default value of "state.backend.rocksdb.checkpoint.transfer.thread.num"

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.13.0
    • Labels:
      None

      Description

      The default value for the number of threads used to download state artifacts from checkpoint storage should be increased.

      The increase should not pose risk of regression, but does in many cases speed up checkpoint recovery significantly.

      Something similar was reported in this blog post, item (3).
      https://engineering.contentsquare.com/2021/ten-flink-gotchas/

      A default value of 8 (eight) sounds like a good default. It should not result in excessive thread explosion, and already speeds up recovery.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                sewen Stephan Ewen
                Reporter:
                sewen Stephan Ewen
              • Votes:
                0 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: