[FLINK-34984] FLIP-423: Disaggregated State Storage and Management (Umbrella FLIP)

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: API / Core, API / DataStream, Runtime / Checkpointing, Runtime / State Backends
• Assignee: Zakelly Lan
• Reporter: Yuan Mei
• Created: 01/Apr/24 11:46
• Updated: 08/Oct/24 10:32

Description

The past decade has witnessed a dramatic shift in Flink's deployment mode, workload patterns, and hardware improvements. We've moved from the map-reduce era where workers are computation-storage tightly coupled nodes to a cloud-native world where containerized deployments on Kubernetes become standard. To enable Flink's Cloud-Native future, we introduce Disaggregated State Storage and Management that uses DFS as primary storage in Flink 2.0

This new architecture is aimed to solve the following challenges brought in the cloud-native era for Flink.
1. Local Disk Constraints in containerization
2. Spiky Resource Usage caused by compaction in the current state model
3. Fast Rescaling for jobs with large states (hundreds of Terabytes)
4. Light and Fast Checkpoint in a native way

 

Design Details can be found in FLIP-423

Proposed changes can be found here:

• Asynchronous State APIs (FLIP-424)
• Non-blocking Asynchronous Execution Model: Parallel I/O (FLIP-425)
• Batching for Network I/O: Beyond Parallel I/O (FLIP-426)
• Disaggregated State Store: ForSt (FLIP-427)
• Faster Checkpoint/Restore/Rescale: Leverage Shared DFS (FLIP-428)

Issue Links

• is a parent of: FLINK-34973 FLIP-425: Asynchronous Execution Model
• is a parent of: FLINK-34981 FLIP-426: Grouping Remote State Access
• is a parent of: FLINK-34982 FLIP-428: Fault Tolerance/Rescale Integration for Disaggregated State
• is a parent of: FLINK-34974 FLIP-424: Asynchronous State APIs
• is a parent of: FLINK-34975 FLIP-427: ForSt - Disaggregated State Store

Sub-Tasks

• Documentation for Disaggregated State Storage and Management: Open