Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-2478

KafkaConsumer javadoc example seems wrong

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.9.0.0
    • Fix Version/s: 0.9.0.1
    • Component/s: consumer
    • Labels:
      None

      Description

      I was looking at this KafkaConsumer example in the javadoc:
      https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java#L199

      As I understand, commit() method commits the maximum offsets returned by the most recent invocation of poll() method.

      In this example, there's a danger of losing the data.
      Imagine the case where 300 records are returned by consumer.poll()
      The commit will happen after inserting 200 records into the database.
      But it will also commit the offsets for 100 records that are still unprocessed.

      So if consumer fails before buffer is dumped into the database again,
      then those 100 records will never be processed.

      If I'm wrong, could you please clarify the behaviour of commit() method?

        Attachments

          Activity

            People

            • Assignee:
              devstr Dmitry Stratiychuk
              Reporter:
              devstr Dmitry Stratiychuk
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: