Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-10342

Kafka duplicate topic consumption when topic name is changed

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Hide
      Starting from 1.8.0, the {{FlinkKafkaConsumer}} now always filter out restored partitions that are no longer associated with a specified topic to subscribe to in the restored execution. This behaviour did not exist in previous versions of the {{FlinkKafkaConsumer}}. If you wish to retain the previous behaviour, please use the {{disableFilterRestoredPartitionsWithSubscribedTopics}} configuration method on the {{FlinkKafkaConsumer}}.
      Show
      Starting from 1.8.0, the {{FlinkKafkaConsumer}} now always filter out restored partitions that are no longer associated with a specified topic to subscribe to in the restored execution. This behaviour did not exist in previous versions of the {{FlinkKafkaConsumer}}. If you wish to retain the previous behaviour, please use the {{disableFilterRestoredPartitionsWithSubscribedTopics}} configuration method on the {{FlinkKafkaConsumer}}.

    Description

      In case of topic name is simply renamed for a KafkaConsumer Flink starts to consume from old and a new topic in the same time which can lead to unexpected behavior.

      Here is the PR with reproduce: https://github.com/apache/flink/pull/6691

       

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            f.li Feng LI
            Oleksandr Nitavskyi Oleksandr Nitavskyi
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 10m
                10m

                Slack

                  Issue deployment