Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-33127

HeapKeyedStateBackend: use buffered I/O to speed up local recovery

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • Runtime / State Backends
    • None

    Description

      Recently, I observed a slow restore case in local recovery using hashmap statebackend.

      It took 147 seconds to restore from a 467MB snapshot, 9 times slower than that (16s) when restore from remote fs.

      The thread dump show that It read local snapshot file directly by unbuffered FileInputStream / fs.local.LocalDataInputStream.

      Maybe we can wrap with BufferInputStream to speed up local recovery.

      Attachments

        1. thread_dump.png
          1.20 MB
          Yangyang ZHANG

        Issue Links

          duplicates

          Improvement - An improvement or enhancement to an existing feature or task. FLINK-26586 FileSystem uses unbuffered read I/O

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. FLINK-19911 add read buffer for input stream

          • Not a Priority -
          • Closed

          Activity

            People

              zhangyy91 Yangyang ZHANG
              zhangyy91 Yangyang ZHANG
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated: