Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-7195

FlinkKafkaConsumer should not respect fetched partitions to filter restored partition states

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 1.3.1
    • Fix Version/s: 1.3.2
    • Component/s: Kafka Connector
    • Labels:
      None

      Description

      This issue is a re-appearance of FLINK-6006. On restore, we should not respect any fetched partitions list from Kafka and perform any filtering of the restored partition states. There are corner cases where, due to Kafka broker downtime, some partitions may be missing in the fetched partition list. To be more precise, we actually should not require fetching partitions on restore.

      We've stepped on our own foot again and reintroduced this bug in https://github.com/apache/flink/pull/3378/commits/ed68fedbe90db03823d75a020510ad3c344fa73e. The previous test for this behavior was too implementation specific, and therefore the leak in catching this on different internal implementations.
      We should have a proper unit test for this that does not rely on the internal implementations and test only on public abstractions of FlinkKafkaConsumerBase.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                tzulitai Tzu-Li (Gordon) Tai
                Reporter:
                tzulitai Tzu-Li (Gordon) Tai
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: