[KAFKA-15470] Allow creating connectors in a stopped state

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.7.0
• Component/s: connect
• Labels: connect kafka-connect kip-required
• Assignee: Yash Mayya
• Reporter: Yash Mayya
• Created: 15/Sep/23 11:46
• Updated: 26/Dec/23 16:55
• Resolved: 15/Nov/23 06:08

Description

KIP-875: First-class offsets support in Kafka Connect introduced a new STOPPED state for connectors along with some REST API endpoints to retrieve and modify offsets for connectors. Currently, only connectors that already exist can be stopped and any newly created connector will always be in the RUNNING state initially. Allowing the creation of connectors in a STOPPED state will facilitate multiple new use cases. One interesting use case would be to migrate connectors from one Kafka Connect cluster to another. Individual connector migration would be useful in a number of scenarios such as breaking a large cluster into multiple smaller clusters (or vice versa), moving a connector from a cluster running in one data center to another etc. A connector migration could be achieved by using the following sequence of steps :-

1. Stop the running connector on the original Kafka Connect cluster
2. Retrieve the offsets for the connector via the GET /connectors/{connector}/offsets  endpoint
3. Create the connector in a stopped state using the same configuration on the new Kafka Connect cluster
4. Alter the offsets for the connector on the new cluster via the PATCH /connectors/{connector}/offsets  endpoint (using the offsets obtained from the original cluster)
5. Resume the connector on the new cluster and delete it on the original cluster

Another use case for creating connectors in a stopped state could be deploying connectors as a part of a larger data pipeline before the source / sink data system has been created or is ready for data transfer.

Issue Links

• links to: GitHub Pull Request #14704
• mentioned in: Page Loading...