[IGNITE-19102] CDC: Use manual assignment for consumer in KafkatoIgniteMetadataUpdater

• Type: Task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: extensions
• Labels: IEP-59 ise
• Ignite Flags: Docs Required, Release Notes Required
• Assignee: Unassigned
• Reporter: Ilya Shishkov
• Created: 22/Mar/23 12:58
• Updated: 24/Nov/23 07:26

Description

Currently, KafkaToIgniteMetadataUpdater uses KafkaConsumer#subscribe, which lead to utilization of the consumer group management [1, "Consumer Groups and Topic Subscriptions"]. Kafka group management has some possible side-effects, which can be avoided by means of manual assignment:

1. KafkaConsumer leaves consumer group after max.poll.interval.ms exceeds [2], because KafkaToIgniteMetadataUpdater polls data only on demand, not in continuous manner.
2. Delays and failures during consumer group rebalancing.
3. Metadata topic should have single partition. Multiple consumers from the same group can not subscribe on the same partition.

Important: KafkaConsumer in KafkaToIgniteMetadataUpdater must be subscribed / assigned to all partitions.

Links:

1. https://kafka.apache.org/27/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html
2. https://kafka.apache.org/27/documentation.html#consumerconfigs_max.poll.interval.ms

Issue Links

• relates to: IGNITE-18209 Reduce binary metadata synchronization time for CDC through Kafka