Details
-
Wish
-
Status: Resolved
-
Major
-
Resolution: Abandoned
-
None
-
None
-
None
Description
The context :
I have :
- only one type of message per topic
- the same number of consumers and topics
- each consumer subscribes to all topics in the same microservice
- a strategy where I stopped the consumer if the consumption failed
The need :
I would like to have a Topic Round Robin Assignor in order to assign all partitions of same topic to exactly one consumer, therefore I will be able to continue the consumption of one topic even if one failed.
If there are exactly the same number of consumers and topics, then each consumer will get all partitions of one topic.
If there are more consumers than topics, then some consumer will not have any partitions to consume.
If there are less consumers than topics, then some consumer will have multiple topics to consume.
As far as I know, there are currently 4 different strategies : CooperativeStickyAssignor, RangeAssignor, RoundRobinAssignor, StickyAssignor.
Therefore, I have written my own Topic Round Robin Assignor that assigns all partitions from each topic to exactly one consumer.
For example, suppose there are two consumers C0 and C1, two topics t0 and t1, and each topic has 3 partitions, resulting in partitions t0p0, t0p1, t0p2, t1p0, t1p1, and t1p2.
The assignment will be:
C0: [t0p0, t0p1 t0p1]
C1: [t1p0, t1p1, t1p2]
First of all, I would like to know if this is a legitimate need.
If this is the case, if you are interested to have a Pull Request about it.
Thank you in advance.
Attachments
Issue Links
- links to