Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-24347

KafkaSource cannot checkpoint if the parallelism is higher than the partition number

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

      The KafkaSourceEnumerator signals all the readers if there are no more splits available and the readers shut down. In case the parallelism is higher than the partitions of the consumed topic there are not enough partitions to distribute and reader subtasks go immediately into FINISHED state.
      Currently, it is not possible to checkpoint if parts of the job are finished. (this is lifted once FLIP-147 is by default enabled Flink 1.15+)

      We should only signal to the readers to go into finished if the job is in bounded execution mode and keep them idling otherwise.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            fpaul Fabian Paul
            fpaul Fabian Paul
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Issue deployment