[KAFKA-2172] Round-robin partition assignment strategy too restrictive

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.10.2.0
• Assignee: Andrew Olson
• Reporter: Jason Rosenberg
• Created: 05/May/15 22:35
• Updated: 01/Feb/17 21:03
• Resolved: 01/Feb/17 21:03

Description

The round-ropin partition assignment strategy, was introduced for the high-level consumer, starting with 0.8.2.1. This appears to be a very attractive feature, but it has an unfortunate restriction, which prevents it from being easily utilized. That is that it requires all consumers in the consumer group have identical topic regex selectors, and that they have the same number of consumer threads.

It turns out this is not always the case for our deployments. It's not unusual to run multiple consumers within a single process (with different topic selectors), or we might have multiple processes dedicated for different topic subsets. Agreed, we could change these to have separate group ids for each sub topic selector (but unfortunately, that's easier said than done). In several cases, we do at least have separate client.ids set for each sub-consumer, so it would be incrementally better if we could at least loosen the requirement such that each set of topics selected by a groupid/clientid pair are the same.

But, if we want to do a rolling restart for a new version of a consumer config, the cluster will likely be in a state where it's not possible to have a single config until the full rolling restart completes across all nodes. This results in a consumer outage while the rolling restart is happening.

Finally, it's especially problematic if we want to canary a new version for a period before rolling to the whole cluster.

I'm not sure why this restriction should exist (as it obviously does not exist for the 'range' assignment strategy). It seems it could be made to work reasonably well with heterogenous topic selection and heterogenous thread counts. The documentation states that "The round-robin partition assignor lays out all the available partitions and all the available consumer threads. It then proceeds to do a round-robin assignment from partition to consumer thread."

If the assignor can "lay out all the available partitions and all the available consumer threads", it should be able to uniformly assign partitions to the available threads. In each case, if a thread belongs to a consumer that doesn't have that partition selected, just move to the next available thread that does have the selection, etc.