Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-4040

Round Robin Assignment does not create balanced assignments

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 0.10.0.0, 0.11.0.0, 1.0.0
    • Fix Version/s: None
    • Component/s: replication
    • Labels:
      None

      Description

      The RoundRobinAssignor loops over the consumers in a circular manner, assigning each time N partitions to each consumer, where N is the number of consuming threads.

      In the following scenario, this creates a major imbalance in the assignment.

      single topic with 64 partitions, 4 consumers (A,B,C,D), each with 12 threads. The roundrobin strategy will allocate the partitions in the following manner:

      A: 24
      B: 16
      C: 12
      D: 12

      when the expected assignment would rather be 16 to each consumer.

      The reason for this imbalance is that instead of allocating a single partition to each consumer in sequence, the assignor attempts to assign 12 partitions each time it is handed a consumer by the circularIterator, so it starts by assigning 12 to each of A,B,C and D, then it has 16 partitions left and it is handed A again, to which it assigns 12 partitions before moving to B to which 4 will be assigned.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              herberts Mathias Herberts
            • Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: