Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-10009

Add method for getting last record offset in kafka partition

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • clients, consumer
    • None

    Description

      As far as I understand, at the current moment, there is no reliable way for getting offset of the last record in the partition using java client. There is endOffsets method in the consumer. And usually endOffsets - 1 works fine. But in the case of transactional producer, topic may contain offsets without a record. And endOffsets - 1 will point to the offset without record.

      This feature will help in situations when consumer application wants to consume the whole topic. Checking of beginning and last record offset will give lower and upper bounds for consuming. Of course, it is doable with the current consumer implementation, but I need to check position after each poll.

      Also, I believe that this feature may help with monitoring and operations.

      Attachments

        Activity

          People

            Unassigned Unassigned
            lmnet Yuriy Badalyantc
            Votes:
            2 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated: