Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-7835

Kafka Connect - Unable to create multiple connectors in a single HTTP (REST) call

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.1.1
    • Fix Version/s: None
    • Component/s: KafkaConnect

      Description

      We have 6 connectors under distributed Kafka worker. Each of these connectors has 18-36 tasks. Each time we add new connector all existing connectors restarts/re-balance. 

      Issue:

      • For each new connector we add, previous/existing connector(s) restarts along with new connector.
      • All existing connectors and their tasks goes into re-balance mode.

      Assuming that we want to add 3 new connectors in one flow or call, how can we do that in one HTTP call rather than 3 HTTP POST calls. Because after each HTTP call we have to wait for re-balance to finish and wait for all tasks to be up again across all instances. 

      Is there any way to create all these 3 connectors for same distributed worker in one HTTP call?

      If not, can we add this as a improvement?

      Something like

      ##POST http://localhost:8083/connectors
      [
        {
           "name": "connector-1",
           "config": {
                "connector.class": "Connector1Class",
                "tasks.max": "36",
                "run.next.start": "5",
                "fetchsize": "100000",
                "topic": "message.connector-1",
                "offset.start": "0"
           }
        },
        {
            "name": "connector-2",
            "config": {
                 "connector.class": "Connector2Class",
                 "tasks.max": "18",
                 "run.next.start": "5",
                 "fetchsize": "100000",
                 "topic": "message.connector-2",
                 "offset.start": "0"
            }
        },
        {
            "name": "connector-3",
            "config": {
                 "connector.class": "Connector3Class",
                 "tasks.max": "18",
                 "run.next.start": "5",
                 "fetchsize": "100000",
                 "topic": "message.connector-3",
                 "offset.start": "0"
            }
        }
      ]
      

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              suraj_f Suraj Fale
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: