[FLINK-21099] Introduce JobType to distinguish between batch and streaming jobs

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.13.0
• Fix Version/s: 1.13.0
• Component/s: Runtime / Coordination
• Labels: pull-request-available
• Assignee: Robert Metzger
• Reporter: Till Rohrmann
• Created: 22/Jan/21 15:22
• Updated: 28/May/21 08:15
• Resolved: 29/Jan/21 12:53

Description

In order to distinguish between batch and streaming jobs we propose to introduce an enum JobType which is set in the JobGraph when creating it. Using the JobType it will be possible to decide which scheduler to use depending on the nature of the job.

For batch jobs (from the DataSet API), setting this field is trivial (in the JobGraphGenerator).

For streaming jobs the situation is more complicated, since FLIP-134 introduced support for bounded (batch) jobs in the DataStream API. For the DataStream API, we rely on the result of StreamGraphGenerator#shouldExecuteInBatchMode, which checks if the DataStream program has unbounded sources.

Lastly, the Blink Table API / SQL Planner also generates StreamGraph instances, which contain batch jobs. We are tagging the StreamGraph as a batch job in the ExecutorUtils.setBatchProperties() method.

Issue Links

• links to: GitHub Pull Request #14767