[KAFKA-5505] Connect: Do not restart connector and existing tasks on task-set change

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.10.2.1
• Fix Version/s: 2.3.0
• Component/s: connect
• Assignee: Konstantine Karantasis
• Reporter: Per Steffensen
• Reviewer: Randall Hauch
• Created: 23/Jun/17 08:07
• Updated: 14/Feb/20 01:29
• Resolved: 17/May/19 03:48

Description

I am writing a connector with a frequently changing task-set. It is really not working very well, because the connector and all existing tasks are restarted when the set of tasks changes. E.g. if the connector is running with 10 tasks, and an additional task is needed, the connector itself and all 10 existing tasks are restarted, just to make the 11th task run also. My tasks have a fairly heavy initialization, making it extra annoying. I would like to see a change, introducing a "mode", where only new/deleted tasks are started/stopped when notifying the system that the set of tasks changed (calling context.requestTaskReconfiguration() - or something similar).

Discussed this issue a little on [e-mail] in the thread "Kafka Connect: To much restarting with a SourceConnector with dynamic set of tasks"

Issue Links

• fixes: KAFKA-7381 Parameterize connector rebalancing behavior
• is duplicated by: KAFKA-7644 Worker Re balance enhancements
• is related to: KAFKA-6433 Connect distributed workers should fail if their config is "incompatible" with leader's
• is related to: KAFKA-6440 Expose Connect leader via REST
• links to: GitHub Pull Request #6363