[IGNITE-19458] Kafka Connect IgniteSinkConnector multiple topics & caches do not work

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.15
• Component/s: extensions
• Ignite Flags: Docs Required, Release Notes Required
• Assignee: Unassigned
• Reporter: Vedran Ljubovic
• Created: 11/May/23 12:45
• Updated: 06/Jun/23 10:17

Description

We are using Kafka Connect to stream 4 topics into 4 separate caches. We are presently forced to use 4 separate instances of Kafka Connect because when 4 connectors are passed to one instance all messages will be streamed into the same (first) Ignite cache. Multiple Connect instances work, but in the future we plan to add more topics/caches and this will affect memory usage as well as our ability to distribute workload in a cluster.

I think that the reason for this is in IgniteSinkTask.java where IgniteDataStreamer is declared as private static final attribute of inner class, which means that it will be initialized with the first passed cache and cannot be changed later.

I'm not sure which is the best approach performance wise, but possible solutions would be:

• It should be non-static attribute of IgniteSinkTask (different streamer for each task)
• It should be created in IgniteSinkConnector and passed to IgniteSinkTask (one streamer per connector)
• Have a Map of cacheNames to Streamers which ensures that only as many Streamers will be created as there are different caches used.

Also I notice that cacheName, SingleTupleExtractor, stopped boolean, are all declared as static which cause various problems when using multiple connectors (messages being processed with wrong SimpleTupleExtractor etc.)

Issue Links

• links to: GitHub Pull Request #215