Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-26267

Kafka source may reprocess data

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

      Due to KAFKA-7703, when the Kafka source tries to get the latest offset, it may get an earliest offset, and then it will reprocess messages that have been processed when it gets the correct latest offset in the next batch.

      This usually happens when restarting a streaming query.

      Attachments

        Issue Links

        is caused by

        Bug - A problem which impairs or prevents the functions of the product. KAFKA-7703 KafkaConsumer.position may return a wrong offset after "seekToEnd" is called

        • Blocker - Blocks development and/or testing work, production could not run
        • Resolved
        is duplicated by

        Improvement - An improvement or enhancement to an existing feature or task. SPARK-28641 MicroBatchExecution committed offsets greater than available offsets

        • Major - Major loss of function.
        • Closed
        is related to

        Improvement - An improvement or enhancement to an existing feature or task. SPARK-28174 Upgrade to Kafka 2.3.0

        • Major - Major loss of function.
        • Resolved
        links to

        Web Link GitHub Pull Request #23324

        Web Link GitHub Pull Request #23365

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            zsxwing Shixiong Zhu
            zsxwing Shixiong Zhu
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment