Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-6006

Kafka Consumer can lose state if queried partition list is incomplete on restore

    Details

      Description

      In 1.1.x and 1.2.x, the FlinkKafkaConsumer performs partition list querying on restore. Then, only restored state of partitions that exists in the queried list is used to initialize the fetcher's state holders.

      If in any case the returned partition list is incomplete (i.e. missing partitions that existed before, perhaps due to temporary ZK / broker downtime), then the state of the missing partitions is dropped and cannot be recovered anymore.

      In 1.3-SNAPSHOT, this is fixed by changes in FLINK-4280, so only 1.1 and 1.2 is affected.

      We can backport some of the behavioural changes there to 1.1 and 1.2. Generally, we should not depend on the current partition list in Kafka when restoring, but just restore all previous state into the fetcher's state holders.

      This would therefore also require some checking on how the consumer threads / Kafka clients behave when its assigned partitions cannot be reached.

        Attachments

          Activity

            People

            • Assignee:
              tzulitai Tzu-Li (Gordon) Tai
              Reporter:
              tzulitai Tzu-Li (Gordon) Tai
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: