Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-1006

Consumer loses messages of a new topic with auto.offset.reset = largest

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Auto Closed
    • 0.8.0
    • None
    • None

    Description

      Consumer currently uses auto.offset.reset = largest by default. If a new topic is created, consumer's topic watcher is fired. The consumer will first finish partition reassignment as part of rebalance and then start consuming from the tail of each partition. Until the partition reassignment is over, the server may have appended new messages to the new topic, consumer won't consume these messages. Thus, multiple batches of messages may be lost when a topic is newly created.

      The fix is to start consuming from the earliest offset for newly created topics.

      Attachments

        Activity

          People

            guozhang Guozhang Wang
            swapnilghike Swapnil Ghike
            Votes:
            4 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: