[KAFKA-5451] Kafka Connect should scan classpath asynchronously

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.11.0.0
• Component/s: connect
• Labels: performance
• Assignee: Randall Hauch
• Reporter: Randall Hauch
• Created: 14/Jun/17 21:53
• Updated: 22/Feb/23 23:14

Description

When Kafka Connect workers start up, they scan the classpath and module paths for connectors, transformations, and converters. This takes anywhere from 15-30sec or longer depending upon how many JARs are included. Currently, this scanning is done synchronously during startup of the Kafka Connect workers, even though the workers may not need the result of the scan.

The scanning logic should be asynchronous and should only block any components that require the result of the scan. This will improve startup time of the workers.

Issue Links

• is related to: KAFKA-6208 Reduce startup time for Kafka Connect workers
• relates to: KAFKA-5450 Scripts to startup Connect in system tests have too short a timeout
• relates to: KAFKA-14627 Modernize Connect plugin discovery