Details
-
New Feature
-
Status: In Progress
-
Major
-
Resolution: Unresolved
-
None
Description
This is a sub-FLIP for the disaggregated state management and its related work, please read the FLIP-423 first to know the whole story.
To maximize I/O capacity utilization and enhance the use of pre-allocated computational resources, this FLIP proposes the introduction of asynchronous state APIs. These APIs permit state access to be executed in threads separate from the task thread, returning the result when available. Consequently, the task thread can process another element while awaiting multiple pending state results. This enables concurrent processing of multiple records, ensuring that the latency of individual I/O operations no longer has a direct impact on job performance. This approach is particularly advantageous in scenarios where I/O bandwidth is underutilized and I/O latency is the limiting factor. The Disaggregated Storage Architecture, as discussed in FLIP-423, is a prime example of a scenario characterized by abundant and easily scalable I/O bandwidth coupled with higher I/O latency. The asynchronous state APIs hold great promise for significantly enhancing Flink's performance when dealing with disaggregated state.
Attachments
Issue Links
- is a child of
-
-
- Open
-
- mentioned in
-
Page Loading...