[FLINK-28130] FLIP-224: Blocklist Mechanism

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Done
• Affects Version/s: 1.16.0
• Fix Version/s: 1.16.0
• Component/s: Runtime / Coordination
• Assignee: Lijie Wang
• Reporter: Lijie Wang
• Created: 20/Jun/22 11:14
• Updated: 05/Jun/23 11:31
• Resolved: 16/Aug/22 03:57

Description

In order to support speculative execution for batch jobs(FLIP-168), we need a mechanism to block resources on nodes where the slow tasks are located. We propose to introduce a blocklist mechanism as follows:  Once a node is marked as blocked, future slots should not be allocated from the blocked node, but the slots that are already allocated will not be affected.

More details see FLIP-224

This is the umbrella ticket to track all the changes of this feature.

Issue Links

• is related to: FLINK-32253 Blocklist unblockResources does not update the pending resource request
• is required by: FLINK-28131 FLIP-168: Speculative Execution for Batch Job

Sub-Tasks

• Enrich TaskManagerLocation with node information: Closed
• Introduce BlocklistHandler: Closed
• Let JobMaster support blocklist mechanism: Closed
• Let ResourceManager support blocklist mechanism: Closed
• Sync blocklist information between JobMaster & ResourceManager: Closed
• Let BlocklistDeclarativeSlotPool accept duplicate slot offers: Closed
• Simplify logs of blocklist: Closed
• Notify all newlyAdded/Merged blocked nodes to BlocklistListener: Closed