Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-687

Rebalance algorithm should consider partitions from all topics

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.10.1.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      The current rebalance step, as stated in the original Kafka paper [1], splits the partitions per topic between all the consumers. So if you have 100 topics with 2 partitions each and 10 consumers only two consumers will be used. That is, for each topic all partitions will be listed and shared between the consumers in the consumer group in order (not randomly).

      If the consumer group is reading from several topics at the same time it makes sense to split all the partitions from all topics between all the consumer. Following the example, we will have 200 partitions in total, 20 per consumer, using the 10 consumers.

      The load per topic could be different and the division should consider this. However even a random division should be better than the current algorithm while reading from several topics and should harm reading from a few topics with several partitions.

        Attachments

        1. KAFKA-687.patch
          38 kB
          Joel Koshy
        2. KAFKA-687_2014-08-28_16:20:25.patch
          36 kB
          Joel Koshy
        3. KAFKA-687_2014-08-25_12:36:48.patch
          42 kB
          Joel Koshy
        4. KAFKA-687_2014-08-20_18:09:28.patch
          45 kB
          Joel Koshy
        5. KAFKA-687_2014-08-19_12:07:37.patch
          40 kB
          Joel Koshy
        6. KAFKA-687_2014-07-18_15:55:15.patch
          228 kB
          Joel Koshy

          Issue Links

            Activity

              People

              • Assignee:
                jjkoshy Joel Koshy
                Reporter:
                pbg Pablo Barrera
                Reviewer:
                Jun Rao
              • Votes:
                3 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: