Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-21075 FLIP-160: Adaptive scheduler
  3. FLINK-21099

Introduce JobType to distinguish between batch and streaming jobs

    XMLWordPrintableJSON

    Details

      Description

      In order to distinguish between batch and streaming jobs we propose to introduce an enum JobType which is set in the JobGraph when creating it. Using the JobType it will be possible to decide which scheduler to use depending on the nature of the job.

      For batch jobs (from the DataSet API), setting this field is trivial (in the JobGraphGenerator).

      For streaming jobs the situation is more complicated, since FLIP-134 introduced support for bounded (batch) jobs in the DataStream API. For the DataStream API, we rely on the result of StreamGraphGenerator#shouldExecuteInBatchMode, which checks if the DataStream program has unbounded sources.

      Lastly, the Blink Table API / SQL Planner also generates StreamGraph instances, which contain batch jobs. We are tagging the StreamGraph as a batch job in the ExecutorUtils.setBatchProperties() method.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                rmetzger Robert Metzger
                Reporter:
                trohrmann Till Rohrmann
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: