[FLINK-26547] Accepting slots without a matching requirement leads to unfulfillable requirements

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.15.0
• Fix Version/s: 1.15.0
• Component/s: Runtime / Coordination
• Labels: pull-request-available
• Assignee: Chesnay Schepler
• Reporter: Chesnay Schepler
• Created: 09/Mar/22 13:31
• Updated: 11/Mar/22 13:35
• Resolved: 11/Mar/22 13:35

Description

To allow recovered TMs to eagerly re-offer their slots we allowed the registration of slots without a matching requirement if the job is currently restarting.

All slots that the pool accepts are mapped to a certain requirement, in order to determine whether sufficient slots were received yet. If a slot is reserved for a requirement that does not coincide with the mapping the pool come up with, then the mapping and requirements are changed accordingly to ensure we still request sufficient slots.

This leads to issues with slots that were accepted without a matching requirement. Those were mapped to the actual resource profile of the slot (to fit into the book-keeping). With the above logic in place this could lead to a specific resource requirement being added, which the remaining JM components are not aware of (and thus will never get rid of).

Issue Links

• is caused by: FLINK-25855 DefaultDeclarativeSlotPool rejects offered slots when the job is restarting
• relates to: FLINK-26274 Test local recovery works across TaskManager process restarts
• links to: GitHub Pull Request #19019