[FLINK-34974] FLIP-424: Asynchronous State APIs

• Type: New Feature
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Fix Version/s: 2.0.0
• Component/s: Runtime / State Backends
• Assignee: Zakelly Lan
• Reporter: Zakelly Lan
• Created: 01/Apr/24 03:40
• Updated: 01/Apr/24 11:52

Description

This is a sub-FLIP for the disaggregated state management and its related work, please read the FLIP-423 first to know the whole story.

To maximize I/O capacity utilization and enhance the use of pre-allocated computational resources, this FLIP proposes the introduction of asynchronous state APIs. These APIs permit state access to be executed in threads separate from the task thread, returning the result when available. Consequently, the task thread can process another element while awaiting multiple pending state results. This enables concurrent processing of multiple records, ensuring that the latency of individual I/O operations no longer has a direct impact on job performance. This approach is particularly advantageous in scenarios where I/O bandwidth is underutilized and I/O latency is the limiting factor. The Disaggregated Storage Architecture, as discussed in FLIP-423, is a prime example of a scenario characterized by abundant and easily scalable I/O bandwidth coupled with higher I/O latency. The asynchronous state APIs hold great promise for significantly enhancing Flink's performance when dealing with disaggregated state.

Issue Links

• is a child of: FLINK-34984 FLIP-423: Disaggregated State Storage and Management (Umbrella FLIP)
• mentioned in: Page Loading...

Sub-Tasks

• Introduce Asynchronous State APIs: Resolved
• Implement State Future and related utilities: Resolved
• Introduce Internal State Interface for Async State API: Resolved
• Introduce New KeyedStateBackend related Async interfaces: Resolved
• Internal Async State Implementation and StateDescriptor for Map/List State: Resolved
• Basic State Iterator for async processing: Resolved
• Create State V2 from new StateDescriptor: Open
• Support TTL for Async State API: Resolved
• Async aggregating state: Resolved
• Async reducing state: Resolved
• Namespace support for async state: Resolved
• Add synchronous interfaces for State V2 API.: Resolved
• Implement AsyncKeyedStateBackend for RocksDBKeyedStateBackend and HeapKeyedStateBackend.: Resolved