[KAFKA-14670] Refactor connect plugins to be called from wrapper classes

• Type: Improvement
• Status: In Progress
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 3.5.0
• Component/s: connect
• Assignee: Greg Harris
• Reporter: Greg Harris
• Created: 01/Feb/23 21:13
• Updated: 02/Mar/23 17:24

Description

Currently, plugin methods in Connect are called directly from the runtime worker code.
This has some advantages:
1. Low (no) overhead: methods are called directly
2. Traceability: searching for usages of an API method or class discovers all of them in the runtime
3. Shorter stacktraces: caller and callee appear together in stacktraces

And some disadvantages:
1. Need to manually ensure that callers swap in the correct ThreadContextClassLoader prior to each call, and swap it back afterwards.
2. Need to manually ensure that callers handle arbitrary exceptions thrown from plugin methods.
3. Need to manually ensure that the caller is safe if the plugin method never returns (infinite loops).

These three disadvantages have appeared in numerous bug fixes over the last several years, some fixing the same core bug but at different call-sites in the runtime code. We can continue to make this tradeoff and pay for it in crashes, incorrect behavior, and development burden, or can attempt to find a holistic solution that solves these problems once, and makes it easier for later feature development and maintenance.

If we introduce a layer of indirection (wrapper classes) around the different plugins, then we can solve some problems in a central place, or tweak the internal signatures to encourage fewer bugs in the caller code.

Issue Links

• relates to: KAFKA-2713 Copycat worker should not call connector's/task's start methods in the control thread
• relates to: KAFKA-4042 DistributedHerder thread can die because of connector & task lifecycle exceptions
• relates to: KAFKA-8340 ServiceLoader fails when used from isolated plugin path directory
• relates to: KAFKA-8819 Plugin path for converters not working as intended
• relates to: KAFKA-9374 Worker can be disabled by blocked connectors
• relates to: KAFKA-9708 Connector does not prefer to use packaged classes during configuration
• relates to: KAFKA-10792 Source tasks can block herder thread by hanging during stop
• relates to: KAFKA-13586 ConfigExceptions thrown by FileConfigProvider during connector/task startup crash worker
• relates to: KAFKA-14645 Plugin classloader not used when retrieving connector plugin config defs via REST API
• relates to: KAFKA-14654 Connectors have incorrect Thread Context Classloader during initialization
• links to: GitHub Pull Request #13185