[FLINK-10643] Bubble execution: Resource aware job execution

• Type: New Feature
• Status: Closed
• Priority: Minor
• Resolution: Abandoned
• Component/s: Runtime / Coordination
• Labels: auto-deprioritized-major
• Assignee: Unassigned
• Reporter: JIN SUN
• Created: 22/Oct/18 23:29
• Updated: 30/Apr/21 09:09
• Resolved: 30/Apr/21 09:09

Description

Today Flink support various channels such as pipelined channel and blocking channel. Blocking channel indicate that data need to be persistent in a batch and then it can be consumed later, it also indicate that the downstream task cannot start to process data unless its producer finished and also downstream task will only depends on this intermediate partition instead of upstream tasks.

By leverage this characteristic, Flink already support fine grain-failover which will build a failover region has reduce failover cost.  However, we can leverage this characteristic even more. As described by this paper (VLDB 2018), Bubble Execution not only use this characteristic to implement fine-grain failover, but also use this to balance the resource utilization and job performance. As shown in the paper (also in the following chart), with 50% of the resource, it get 25% (0.75 speedup) average slow down for TPCH benchmark.

This JIRA here is umbrella that try to apply the idea of this paper to FLINK.

Issue Links

• is related to: FLINK-16430 FLIP-119 Pipelined Region Scheduling