Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-26267

Kafka source may reprocess data

    XMLWordPrintableJSON

Details

    Description

      Due to KAFKA-7703, when the Kafka source tries to get the latest offset, it may get an earliest offset, and then it will reprocess messages that have been processed when it gets the correct latest offset in the next batch.

      This usually happens when restarting a streaming query.

      Attachments

        Issue Links

          is caused by

          Bug - A problem which impairs or prevents the functions of the product. KAFKA-7703 KafkaConsumer.position may return a wrong offset after "seekToEnd" is called

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved
          is duplicated by

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-28641 MicroBatchExecution committed offsets greater than available offsets

          • Major - Major loss of function.
          • Closed
          is related to

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-28174 Upgrade to Kafka 2.3.0

          • Major - Major loss of function.
          • Resolved
          links to

          Web Link GitHub Pull Request #23324

          Web Link GitHub Pull Request #23365

          Activity

            People

              zsxwing Shixiong Zhu
              zsxwing Shixiong Zhu
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: