Kafka
  1. Kafka
  2. KAFKA-1075

Consumer will not rebalance upon topic partition change

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: 0.8.0
    • Component/s: None
    • Labels:
      None

      Description

      Due to the watcher and zk data structure mismatch, consumer will not rebalance upon topic partition change. Details:

      ZK data structure for topic partitions:

      /brokers/topics/[topic-name]/partitions/[partition-id]/state

      When new partitions are added, it will change the data in /brokers/topics/[topic-name], however the ZK client is watching on child change of /brokers/topics/[topic-name], which will always be a single child 'partitions'.

      Proposal: add a data change listener, which will trigger rebalance upon data changes on /brokers/topics/[topic-name].

      1. KAFKA-1075.patch
        4 kB
        Guozhang Wang
      2. KAFKA-1075_2013-10-07_10:25:01.patch
        20 kB
        Guozhang Wang
      3. KAFKA-1075_2013-10-07_10:58:40.patch
        4 kB
        Guozhang Wang

        Activity

        Guozhang Wang created issue -
        Guozhang Wang made changes -
        Field Original Value New Value
        Attachment KAFKA-1075.patch [ 12606912 ]
        Hide
        Guozhang Wang added a comment -
        Show
        Guozhang Wang added a comment - Created reviewboard https://reviews.apache.org/r/14497/
        Guozhang Wang made changes -
        Attachment KAFKA-1075_2013-10-07_10:25:01.patch [ 12607185 ]
        Hide
        Guozhang Wang added a comment -
        Show
        Guozhang Wang added a comment - Updated reviewboard https://reviews.apache.org/r/14497/
        Guozhang Wang made changes -
        Attachment KAFKA-1075_2013-10-07_10:58:40.patch [ 12607195 ]
        Hide
        Guozhang Wang added a comment -
        Show
        Guozhang Wang added a comment - Updated reviewboard https://reviews.apache.org/r/14497/
        Hide
        Jun Rao added a comment -

        Thanks for the patch. +1 and committed to 0.8.

        Show
        Jun Rao added a comment - Thanks for the patch. +1 and committed to 0.8.
        Jun Rao made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Fix Version/s 0.8 [ 12317244 ]
        Resolution Fixed [ 1 ]
        Jun Rao made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Resolved Resolved
        2d 23h 6m 1 Jun Rao 07/Oct/13 22:24
        Resolved Resolved Closed Closed
        13s 1 Jun Rao 07/Oct/13 22:24

          People

          • Assignee:
            Guozhang Wang
            Reporter:
            Guozhang Wang
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development