[KAFKA-18119] Service loading loads incorrect plugin version.

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.6.0, 3.7.0, 3.6.1, 3.6.2, 3.8.0, 3.7.1, 3.9.0, 3.8.1
• Component/s: connect
• Assignee: Unassigned
• Reporter: Snehashis Pal
• Created: 29/Nov/24 07:44
• Updated: 02/Dec/24 20:50

Description

Kafka connect seems to be loading incorrect version of connect plugins (such as connectors) if run in SERVICE_LOAD mode. If the worker is started with only service loading enabled, it does not seem to be loading the latest version of a plugin available in its plugins path. Rather it always seems to be defaulting to the plugin version provided in the classpath. 

I observed this when I placed an updated json-converter in the plugins path and a connector instantiated, still defaulted to using the one provided in the connect (Kafka) distribution. It does not happen when the older reflections-based plugin scanner is used. This can be reproduced by following the same process, noted down below

• Set plugin.discovery=SERVICE_LOAD
• Install a newer version of json-converter (org.apache.kafka.connect.json.JsonConverter) than the one provided in the distribution. Usually, the bundled version is the same as the current Kafka distribution. 
• Start a connector with either key.converter or value.converter set to the json converter. Without this in the config the latest version is loaded.

It might be hard to judge which version of the converter is loaded. For testing it might be good to build a new json converter with some log lines depicting the version in use. Another way would be run connect in debug mode and add some breakpoints in the startTask method in Worker where the converters are initialised.

Issue Links

• is related to: KAFKA-6513 New Connect header support doesn't define `converter.type` property correctly
• is related to: KAFKA-8819 Plugin path for converters not working as intended
• is related to: KAFKA-14789 Plugin ServiceLoader visibility from isolated plugins is inconsistent