Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-6467

Enforce layout of dependencies within a Connect plugin to be deterministic

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.11.0.2, 1.0.0
    • Fix Version/s: 0.11.0.3, 1.0.1
    • Component/s: KafkaConnect
    • Labels:
      None

      Description

      In principle, Connect plugins that intend to load their dependencies in isolation should not contain any conflicts among the classes they package as dependencies. In other words, the order in which a plugin's dependencies are laid out and passed to its plugin class loader should not matter.

      However, in practice, there are rare and suboptimal situations where a plugin needs to bundle a few packages with conflicting dependencies because it doesn't control packaging of third-party modules. In those cases depending on a deterministic ordering within the class loader's path can help the Connect plugin enforce loading of the desired classes as needed. (For example, see HDFS connector with MapR libs or HDFS connector with Hive's extended jar)

      To achieve such ordering, this improvement suggests ordering a plugin's dependencies in a nested directory structure by sorting such paths alphanumerically. This way the deterministic order is implicit (no extra configuration is required) and a specific dependency can be put earlier or later in the class loader's path with appropriate naming of its package path (e.g. within the plugin's directory).

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                kkonstantine Konstantine Karantasis
                Reporter:
                kkonstantine Konstantine Karantasis
                Reviewer:
                Jason Gustafson
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: