Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-5479

Per-partition watermarks in FlinkKafkaConsumer should consider idle partitions

Agile BoardRank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersStop watchingWatchersCreate sub-taskLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 1.11.0
    • Connectors / Kafka
    • None

    Description

      Reported in ML: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-topic-partition-skewness-causes-watermark-not-being-emitted-td11008.html

      Similar to what's happening to idle sources blocking watermark progression in downstream operators (see FLINK-5017), the per-partition watermark mechanism in FlinkKafkaConsumer is also being blocked of progressing watermarks when a partition is idle. The watermark of idle partitions is always Long.MIN_VALUE, therefore the overall min watermark across all partitions of a consumer subtask will never proceed.

      It's normally not a common case to have Kafka partitions not producing any data, but it'll probably be good to handle this as well. I think we should have a localized solution similar to FLINK-5017 for the per-partition watermarks in AbstractFetcher.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            aljoscha Aljoscha Krettek
            tzulitai Tzu-Li (Gordon) Tai
            Votes:
            8 Vote for this issue
            Watchers:
            38 Stop watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment