Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-8001

Mark Kafka Consumer as idle if it doesn't have partitions

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Blocker
    • Resolution: Fixed
    • None
    • 1.3.4, 1.4.0
    • None
    • None

    Description

      In Flink 1.3.x the Kafka Consumer will emit a Long.MAX_VALUE watermark if it has zero partitions assigned. If this happens and other parallel instances of the Kafka Consumer are marked as idle (which currently never happens by default but does happen in custom forks of our Kafka code) this means that the watermark jumps to Long.MAX_VALUE downstream.

      In Flink 1.4.x this happens implicitly in the PeriodicWatermarkEmitter in AbstractFetcher where the watermark is Long.MAX_VALUE if we don't have any partitions. This should be changed to mark the source as idle instead, if we don't have any partitions.

      Attachments

        Issue Links

          Activity

            People

              tzulitai Tzu-Li (Gordon) Tai
              aljoscha Aljoscha Krettek
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: