Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-5337

Partition assignment strategy that distributes lag evenly across consumers in each group

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 0.10.2.1
    • Fix Version/s: None
    • Component/s: consumer
    • Labels:
      None

      Description

      Existing partition assignment strategies (RangeAssignor and RoundRobinAssignor) do not account for the current consumer group lag on each partition. This can result in sub-optimal assignments when the distribution of lags for a given topic and consumer group is skewed.

      The LagBasedAssignor operates on a per-topic basis, and attempts to assign partitions such that lag is distributed as evenly across a consumer group.

      Algorithm:

      For each topic, we first obtain the lag on all partitions. Lag on a given partition is the difference between the end offset and the last offset committed by the consumer group. If no offsets have been committed for a partition we determine the lag based on the code auto.offset.reset property. If auto.offset.reset=latest, we assign a lag of 0. If auto.offset.reset=earliest (or any other value) we assume assign lag equal to the total number of message currently available in that partition.

      We then create a map storing the current total lag of all partitions assigned to each member of the consumer group. Partitions are assigned in decreasing order of lag, with each partition assigned to the consumer with least total number of assigned partitions, breaking ties by assigning to the consumer with the least total assigned lag.

      Distributing partitions evenly across consumers (by count) ensures that the partition assignment is balanced when all partitions have a current lag of 0 or if the distribution of lags is heavily skewed. It also gives the consumer group the best possible chance of remaining balanced if the assignment is retained for a long period.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                grantneale Grant Neale
              • Votes:
                2 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated: