[KAFKA-15162] Reflective plugin scanning misses plugins which are in parent classloaders but not classpath

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.6.0
• Component/s: connect
• Assignee: Greg Harris
• Reporter: Greg Harris
• Created: 07/Jul/23 18:18
• Updated: 19/Jul/23 19:50
• Resolved: 19/Jul/23 19:50

Description

When the Plugins class is instantiated with a parent classloader other than the system classloader, as in these tests: https://github.com/apache/kafka/blob/fd5b300b573dc41b94c710a791f9ee6f568992d4/connect/runtime/src/test/java/org/apache/kafka/connect/runtime/isolation/PluginsTest.java#L493-L501

Plugins which are present in that parent classloader but not in the app classloader are not visible. This is because the classes and resources become visible, but the set of URLs used for reflective scanning only includes the classpath jars: https://github.com/apache/kafka/blob/fd5b300b573dc41b94c710a791f9ee6f568992d4/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/isolation/PluginUtils.java#L353 

This causes a discrepancy between the Reflection and ServiceLoading scanners, because the Reflection scanner relies on these URLs but the ServiceLoader relies on the classes and resources instead.

Either the tests should be rewritten to accommodate this difference in behavior, or the PluginUtils.pluginSources should be amended to make a best-effort evaluation of the parent classloader URLs.

Issue Links

• blocks: KAFKA-15031 Add plugin.discovery worker configuration
• Discovered while testing: KAFKA-14627 Modernize Connect plugin discovery
• links to: GitHub Pull Request #13977