Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.8.1
    • Component/s: None
    • Labels:

      Description

      Currently the consumer directly writes their offsets to zookeeper. Two problems with this: (1) This is a poor use of zookeeper, and we need to replace it with a more scalable offset store, and (2) it makes it hard to carry over to clients in other languages. A first step towards accomplishing that is to add a proper Kafka API for committing offsets. The initial version of this would just write to zookeeper as we do today, but in the future we would then have the option of changing this.

      This api likely needs to take a sequence of consumer-group/topic/partition/offset entries and commit them all.

      It would be good to do a wiki design on how this would work and consensus on that first.

      1. KAFKA-657v1.patch
        8 kB
        David Arthur
      2. KAFKA-657v2.patch
        30 kB
        David Arthur
      3. KAFKA-657v3.patch
        30 kB
        David Arthur
      4. KAFKA-657v4.patch
        30 kB
        David Arthur
      5. KAFKA-657v5.patch
        44 kB
        David Arthur
      6. KAFKA-657v6.patch
        45 kB
        David Arthur
      7. KAFKA-657v7.patch
        45 kB
        David Arthur
      8. KAFKA-657v8.patch
        56 kB
        David Arthur

        Issue Links

          Activity

          Jay Kreps created issue -
          Jay Kreps made changes -
          Field Original Value New Value
          Labels project
          David Arthur made changes -
          Attachment KAFKA-657v1.patch [ 12560491 ]
          David Arthur made changes -
          Attachment KAFKA-657v2.patch [ 12560895 ]
          David Arthur made changes -
          Attachment KAFKA-657v3.patch [ 12561350 ]
          David Arthur made changes -
          Attachment KAFKA-657v4.patch [ 12561958 ]
          David Arthur made changes -
          Attachment KAFKA-657v5.patch [ 12562207 ]
          David Arthur made changes -
          Attachment KAFKA-657v6.patch [ 12562478 ]
          David Arthur made changes -
          Attachment KAFKA-657v7.patch [ 12562978 ]
          David Arthur made changes -
          Attachment KAFKA-657v8.patch [ 12563294 ]
          Jay Kreps made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Fix Version/s 0.8.1 [ 12322960 ]
          Resolution Fixed [ 1 ]
          Colin B. made changes -
          Link This issue relates to KAFKA-993 [ KAFKA-993 ]

            People

            • Assignee:
              Unassigned
              Reporter:
              Jay Kreps
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development