Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-15470

Allow creating connectors in a stopped state




      KIP-875: First-class offsets support in Kafka Connect introduced a new STOPPED state for connectors along with some REST API endpoints to retrieve and modify offsets for connectors. Currently, only connectors that already exist can be stopped and any newly created connector will always be in the RUNNING state initially. Allowing the creation of connectors in a STOPPED state will facilitate multiple new use cases. One interesting use case would be to migrate connectors from one Kafka Connect cluster to another. Individual connector migration would be useful in a number of scenarios such as breaking a large cluster into multiple smaller clusters (or vice versa), moving a connector from a cluster running in one data center to another etc. A connector migration could be achieved by using the following sequence of steps :-

      1. Stop the running connector on the original Kafka Connect cluster
      2. Retrieve the offsets for the connector via the GET /connectors/{connector}/offsets  endpoint
      3. Create the connector in a stopped state using the same configuration on the new Kafka Connect cluster
      4. Alter the offsets for the connector on the new cluster via the PATCH /connectors/{connector}/offsets  endpoint (using the offsets obtained from the original cluster)
      5. Resume the connector on the new cluster and delete it on the original cluster

      Another use case for creating connectors in a stopped state could be deploying connectors as a part of a larger data pipeline before the source / sink data system has been created or is ready for data transfer.


        Issue Links



              yash.mayya Yash Mayya
              yash.mayya Yash Mayya
              0 Vote for this issue
              2 Start watching this issue