Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-34973

FLIP-425: Asynchronous Execution Model

    XMLWordPrintableJSON

Details

    Description

      This is a sub-FLIP for the disaggregated state management and its related work, please read the FLIP-423 first to know the whole story.

      FLIP-424 introduces asynchronous state APIs with callbacks allowing state access to be executed in threads separate from the task thread, making better usage of I/O bandwidth and enhancing throughput. This FLIP proposes an execution framework for asynchronous state APIs. The execution code path for the new API is completely independent from the original one, where many runtime components are redesigned. We intend to delve into the challenges associated with asynchronous execution and provide an in-depth design analysis for each module. Furthermore, we will conduct a performance analysis of the new framework relative to the current implementation and examine how it measures up against other potential alternatives.

      Attachments

        Issue Links

          1.
          Basic framework of async execution for state Sub-task Resolved Zakelly Lan
          2.
          Implement record order preservation and buffering of AsyncExecutionController Sub-task Resolved Yanfei Lei
          3.
          Introduce async execution configurations Sub-task Resolved Yanfei Lei
          4.
          Wire AsyncExecutionController to AbstractStreamOperator Sub-task Resolved Zakelly Lan
          5.
          Implement checkpoint drain in AsyncExecutionController Sub-task Resolved Yanfei Lei
          6.
          Timer firing under async execution model Sub-task Resolved Yanfei Lei
          7.
          Wire new operators for async state with DataStream V2 Sub-task Closed Zakelly Lan
          8.
          Store timer in JVM heap when async execution enabled Sub-task Resolved Yanfei Lei
          9.
          Introduce Epoch Manager for async execution Sub-task Resolved Yanfei Lei
          10.
          Latency marker emitting under async execution model Sub-task Resolved Yanfei Lei
          11.
          Error handling in StateFuture's callback Sub-task Resolved Yanfei Lei
          12.
          Bridge between AsyncKeyedStateBackend and AsyncExecutionController Sub-task Resolved Hangxiang Yu
          13.
          Add buffer size/in-flight records metrics for AsyncExecutionController Sub-task Open Unassigned
          14.
          Optimize buffer triggering of async state requests Sub-task Resolved Zakelly Lan
          15.
          Batch execution of async state request callback Sub-task Resolved Zakelly Lan
          16.
          Support object reuse in async state execution Sub-task Open Zakelly Lan
          17.
          Test harness for async state processing operators Sub-task Open Unassigned
          18.
          Directly specify key and process runnable in async state execution model Sub-task Resolved Zakelly Lan
          19.
          Move async state related operators to flink-runtime Sub-task Resolved Zakelly Lan
          20.
          Properly handle KeyContext when using AsyncKeyedStateBackendAdaptor Sub-task Resolved Zakelly Lan
          21.
          More friendly error or warn message for misconfigured statebackend with async state processing Sub-task Resolved Zakelly Lan
          22.
          Decouple the initialization of sync and async keyed statebackend Sub-task Resolved Zakelly Lan
          23.
          Drain state requests after the user function perform snapshotState Sub-task Resolved Zakelly Lan

          Activity

            People

              Yanfei Lei Yanfei Lei
              zakelly Zakelly Lan
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated: