[FLINK-24343] Revisit Scheduler and Coordinator Startup Procedure

• Type: Technical Debt
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.14.0, 1.13.2
• Fix Version/s: 1.20.0
• Component/s: Runtime / Coordination
• Assignee: Unassigned
• Reporter: Stephan Ewen
• Created: 21/Sep/21 08:02
• Updated: 11/Mar/24 12:44

Description

We need to re-examine the startup procedure of the scheduler, and how it interacts with the startup of the operator coordinators.

We need to make sure the following conditions are met:

• The Operator Coordinators are started before the first action happens that they need to be informed of. That includes as task being ready, a checkpoint happening, etc.

• The scheduler must be started to the point that it can handle "failGlobal()" calls, because the coordinators might trigger that during their startup when an exception in "start()" occurs.

/cc chesnay

Issue Links

• is caused by: FLINK-24303 SourceCoordinator exception may fail Session Cluster
• is related to: FLINK-24303 SourceCoordinator exception may fail Session Cluster