[FLINK-26773] ResourceManager leader election can a reconnect while shutting down the JobMaster

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.14.4, 1.15.0, 1.16.0
• Component/s: Runtime / Coordination
• Labels: pull-request-available stale-assigned
• Assignee: Jonathan Lazarus
• Reporter: Matthias Pohl
• Created: 21/Mar/22 10:16
• Updated: 13/Jul/22 22:37

Description

There's a race condition happening with the ResourceManager leader election in the JobMaster while shutting it down. The JobMaster calls dissolveResourceManagerConnection while shutting down itself trying to disconnect itself from the ResourceManager (see JobMaster:1180).

This closes the RM connection to the JobMaster from the ResourceManager's side (see ResourceManager:979. The JobMaster tries to reconnect to the ResourceManager leader if there's still an address stored for that leader (which is the case during shutdown; see JobMaster:790).

The JobMaster shouldn't try to reconnect after it has already freed it's requirements as part of the shutdown.

Issue Links

• is duplicated by: FLINK-27354 JobMaster still processes requests while terminating
• links to: GitHub Pull Request #19729
• links to: GitHub Pull Request #19730