[FLINK-34973] FLIP-425: Asynchronous Execution Model

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Fix Version/s: 2.0.0
• Component/s: Runtime / Checkpointing, Runtime / State Backends, Runtime / Task
• Assignee: Yanfei Lei
• Reporter: Zakelly Lan
• Created: 01/Apr/24 03:36
• Updated: 07/Apr/24 01:58

Description

This is a sub-FLIP for the disaggregated state management and its related work, please read the FLIP-423 first to know the whole story.

FLIP-424 introduces asynchronous state APIs with callbacks allowing state access to be executed in threads separate from the task thread, making better usage of I/O bandwidth and enhancing throughput. This FLIP proposes an execution framework for asynchronous state APIs. The execution code path for the new API is completely independent from the original one, where many runtime components are redesigned. We intend to delve into the challenges associated with asynchronous execution and provide an in-depth design analysis for each module. Furthermore, we will conduct a performance analysis of the new framework relative to the current implementation and examine how it measures up against other potential alternatives.

Issue Links

• is a child of: FLINK-34984 FLIP-423: Disaggregated State Storage and Management (Umbrella FLIP)
• mentioned in: Page Loading...

Sub-Tasks

• Basic framework of async execution for state: Resolved
• Implement record order preservation and buffering of AsyncExecutionController: Resolved
• Introduce async execution configurations: Resolved
• Wire AsyncExecutionController to AbstractStreamOperator: Resolved
• Implement checkpoint drain in AsyncExecutionController: Resolved
• Timer firing under async execution model: Resolved
• Wire new operators for async state with DataStream V2: Resolved
• Store timer in JVM heap when async execution enabled: Resolved
• Introduce Epoch Manager for async execution: Resolved
• Latency marker emitting under async execution model: Resolved
• Error handling in StateFuture's callback: Resolved
• Bridge between AsyncKeyedStateBackend and AsyncExecutionController: Resolved
• Add buffer size/in-flight records metrics for AsyncExecutionController: Open
• Optimize buffer triggering of async state requests: Resolved
• Batch execution of async state request callback: Resolved
• Support object reuse in async state execution: Open
• Test harness for async state processing operators: Open
• Directly specify key and process runnable in async state execution model: Resolved
• Move async state related operators to flink-runtime: Resolved
• Properly handle KeyContext when using AsyncKeyedStateBackendAdaptor: Resolved
• More friendly error or warn message for misconfigured statebackend with async state processing: Resolved
• Decouple the initialization of sync and async keyed statebackend: Resolved