[KAFKA-13136] kafka-connect task.max : active task in consumer group is limited by the bigger topic to consume

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Assignee: Unassigned
• Reporter: raphaelauv
• Created: 26/Jul/21 09:33
• Updated: 17/Mar/22 16:27
• Resolved: 17/Mar/22 16:27

Description

In kafka-connect 2.7

The maximum number of active task for a sink connector is equal to the topic with the biggest number of partitions to consume

An active task is a task with partitions attributed in the consumer-group of the sink connector

example :

With 2 topics where each have 10 partitions ( 20 partitions in total )

The maximum number of active task is 10 ( if I set task.max at 12 ,there is 10 members of the consumer group consuming partitions and  2 members in the consumer-group that do not have partitions to consume).

If I add a third topic with 15 partitions to the connector conf then the 12 members of the consumer group are consuming partitions, and then if I set now task.max at 17 only 15 members are active in the consumer-group.