Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-657

Add an API to commit offsets

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 0.8.1
    • None

    Description

      Currently the consumer directly writes their offsets to zookeeper. Two problems with this: (1) This is a poor use of zookeeper, and we need to replace it with a more scalable offset store, and (2) it makes it hard to carry over to clients in other languages. A first step towards accomplishing that is to add a proper Kafka API for committing offsets. The initial version of this would just write to zookeeper as we do today, but in the future we would then have the option of changing this.

      This api likely needs to take a sequence of consumer-group/topic/partition/offset entries and commit them all.

      It would be good to do a wiki design on how this would work and consensus on that first.

      Attachments

        1. KAFKA-657v8.patch
          56 kB
          David Arthur
        2. KAFKA-657v7.patch
          45 kB
          David Arthur
        3. KAFKA-657v6.patch
          45 kB
          David Arthur
        4. KAFKA-657v5.patch
          44 kB
          David Arthur
        5. KAFKA-657v4.patch
          30 kB
          David Arthur
        6. KAFKA-657v3.patch
          30 kB
          David Arthur
        7. KAFKA-657v2.patch
          30 kB
          David Arthur
        8. KAFKA-657v1.patch
          8 kB
          David Arthur

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            jkreps Jay Kreps
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment