[FLINK-16866] Make job submission non-blocking - ASF JIRA

Agile Board

Rank to Top

Rank to Bottom

Attach files

Attach Screenshot

Bulk Copy Attachments

Bulk Move Attachments

Voters

Watch issue

Watchers

Create sub-task

Convert to sub-task

Link

Clone

Labels

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Improvement
Status: Closed
Priority: Critical
Resolution: Fixed
Affects Version/s: 1.9.2, 1.10.0, 1.11.0
Fix Version/s: 1.12.0
Component/s: Runtime / Coordination
Labels:
- pull-request-available

Release Note:

Hide
The semantics of submitting a job have slightly changed. The submission call returns almost immediately, with the job being in a new INITIALIZING state. Operations such as triggering a savepoint or retrieving the full job details are not available while the job is in that state.
Once the JobManager for that job has been created, the job is in CREATED state and all calls are available.

Show
The semantics of submitting a job have slightly changed. The submission call returns almost immediately, with the job being in a new INITIALIZING state. Operations such as triggering a savepoint or retrieving the full job details are not available while the job is in that state. Once the JobManager for that job has been created, the job is in CREATED state and all calls are available.

Description

Currently, Flink waits to acknowledge a job submission until the corresponding JobManager has been created. Since its creation also involves the creation of the ExecutionGraph and potential FS operations, it can take a bit of time. If the user has configured a too low web.timeout, the submission can time out only reporting a TimeoutException to the user.

I propose to change the notion of job submission slightly. Instead of waiting until the JobManager has been created, a job submission is complete once all job relevant files have been uploaded to the Dispatcher and the Dispatcher has been told about it. Creating the JobManager will then belong to the actual job execution. Consequently, if problems occur while creating the JobManager it will result into a job failure.

Attachments

Issue Links

Add Link

blocks

FLINK-16867 Simplify default timeout configuration

Open

Delete this link

causes

FLINK-19237 LeaderChangeClusterComponentsTest.testReelectionOfJobMaster failed with "NoResourceAvailableException: Could not allocate the required slot within slot request timeout"