[KAFKA-7835] Kafka Connect - Unable to create multiple connectors in a single HTTP (REST) call

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.1.1
• Component/s: connect
• Labels: kafka-connect needs-kip
• Assignee: Unassigned
• Reporter: Suraj Fale
• Created: 17/Jan/19 16:10
• Updated: 22/Jul/19 15:23

Description

We have 6 connectors under distributed Kafka worker. Each of these connectors has 18-36 tasks. Each time we add new connector all existing connectors restarts/re-balance. 

Issue:

• For each new connector we add, previous/existing connector(s) restarts along with new connector.
• All existing connectors and their tasks goes into re-balance mode.

Assuming that we want to add 3 new connectors in one flow or call, how can we do that in one HTTP call rather than 3 HTTP POST calls. Because after each HTTP call we have to wait for re-balance to finish and wait for all tasks to be up again across all instances. 

Is there any way to create all these 3 connectors for same distributed worker in one HTTP call?

If not, can we add this as a improvement?

Something like

##POST http://localhost:8083/connectors
[
  {
     "name": "connector-1",
     "config": {
          "connector.class": "Connector1Class",
          "tasks.max": "36",
          "run.next.start": "5",
          "fetchsize": "100000",
          "topic": "message.connector-1",
          "offset.start": "0"
     }
  },
  {
      "name": "connector-2",
      "config": {
           "connector.class": "Connector2Class",
           "tasks.max": "18",
           "run.next.start": "5",
           "fetchsize": "100000",
           "topic": "message.connector-2",
           "offset.start": "0"
      }
  },
  {
      "name": "connector-3",
      "config": {
           "connector.class": "Connector3Class",
           "tasks.max": "18",
           "run.next.start": "5",
           "fetchsize": "100000",
           "topic": "message.connector-3",
           "offset.start": "0"
      }
  }
]