Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-2491 Checkpointing only works if all operators/tasks are still running
  3. FLINK-21080

Identify JobVertex containing legacy source operators and abort checkpoint with legacy source operators partially finished

    XMLWordPrintableJSON

    Details

      Description

      Most legacy source operators would record the offset for each partitions, and after recovery it would read from the recorded offset. If before a checkpoint some subtasks are finished, the corresponding partition offsets would be deserted in the checkpoint. Then if the job recover with this checkpoint, the legacy source would re-discovery all the partitions and for those finished tasks, the legacy source would re-read them since their offsets are not recorded. 

      Therefore, we would like to fail the checkpoint if some legacy source operators have part of subtasks finished. 

        Attachments

          Activity

            People

            • Assignee:
              maguowei Guowei Ma
              Reporter:
              gaoyunhaii Yun Gao
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: