[FLINK-34007] Flink Job stuck in suspend state after losing leadership in HA Mode

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.19.0, 1.18.1, 1.18.2
• Fix Version/s: 1.19.0
• Component/s: Runtime / Coordination
• Labels: pull-request-available
• Release Note: Fixes a bug where the leader election wasn't able to pick up leadership again after renewing the lease token caused a leadership loss. This required fabric8io:kubernetes-client to be upgraded from v6.6.2 to v6.9.0.
• Assignee: Matthias Pohl
• Reporter: Zhenqiu Huang
• Created: 06/Jan/24 03:35
• Updated: 05/Feb/24 22:10
• Resolved: 05/Feb/24 22:10

Description

The observation is that Job manager goes to suspend state with a failed container not able to register itself to resource manager after timeout.

JM Log, see attached

 

Issue Links

• causes: FLINK-34243 Update fabric8io:kubernetes-client to v6.9.0+ to allow the usage of ReleaseOnCall
• split to: FLINK-34333 Fix FLINK-34007 LeaderElector bug in 1.18
• Testing discovered: FLINK-34343 ResourceManager registration is not completed when registering the JobMaster
• links to: GitHub Pull Request #24132
• links to: GitHub Pull Request #24252
• links to: kubernetes-client Issue #5635
• mentioned in: Page Loading...