Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-3487

KIP-146: Support per-connector/per-task classloaders in Connect

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 0.10.0.0
    • Fix Version/s: 0.11.0.0
    • Component/s: KafkaConnect
    • Labels:

      Description

      Currently we just use the default ClassLoader in Connect. However, this limits how we can compatibly load conflicting connector plugins. Ideally we would use a separate class loader per connector/task that is instantiated to avoid potential conflicts.

      Note that this also opens up options for other ways to provide jars to instantiate connectors. For example, Spark uses this to dynamically publish classes defined in the REPL and load them via URL: https://ardoris.wordpress.com/2014/03/30/how-spark-does-class-loading/ But much simpler examples (include URL in the connector class instead of just class name) are also possible and could be a nice way to more support dynamic sets of connectors, multiple versions of the same connector, etc.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                kkonstantine Konstantine Karantasis
                Reporter:
                ewencp Ewen Cheslack-Postava
                Reviewer:
                Ewen Cheslack-Postava
              • Votes:
                2 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: