[CAMEL-14420] Support `KafkaOffsetBackingStore` in camel-debezim

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.1.0
• Component/s: camel-debezium
• Estimated Complexity: Unknown
• Assignee: Omar Al-Safi
• Reporter: Omar Al-Safi
• Created: 21/Jan/20 15:56
• Updated: 30/Jan/20 06:30
• Resolved: 29/Jan/20 14:08

Description

Currently, when set offsetStorage to org.apache.kafka.connect.storage. KafkaOffsetBackingStore and offsetTopic, the engine fails with the following error:

 
org.apache.kafka.common.config.ConfigException: Missing required configuration "bootstrap.servers" which has no default value.
    at org.apache.kafka.common.config.ConfigDef.parseValue(ConfigDef.java:476) ~[kafka-clients-2.3.1.jar:na]
    at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:466) ~[kafka-clients-2.3.1.jar:na]
    at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:108) ~[kafka-clients-2.3.1.jar:na]
    at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:142) ~[kafka-clients-2.3.1.jar:na]
    at org.apache.kafka.clients.admin.AdminClientConfig.<init>(AdminClientConfig.java:196) ~[kafka-clients-2.3.1.jar:na]
    at org.apache.kafka.clients.admin.AdminClient.create(AdminClient.java:65) ~[kafka-clients-2.3.1.jar:na]
    at org.apache.kafka.connect.util.TopicAdmin.<init>(TopicAdmin.java:169) ~[connect-runtime-2.3.1.jar:na]
    at org.apache.kafka.connect.storage.KafkaOffsetBackingStore$1.run(KafkaOffsetBackingStore.java:98) ~[connect-runtime-2.3.1.jar:na]
    at org.apache.kafka.connect.util.KafkaBasedLog.start(KafkaBasedLog.java:128) ~[connect-runtime-2.3.1.jar:na]
    at org.apache.kafka.connect.storage.KafkaOffsetBackingStore.start(KafkaOffsetBackingStore.java:109) ~[connect-runtime-2.3.1.jar:na]
    at io.debezium.embedded.EmbeddedEngine.run(EmbeddedEngine.java:733) ~[debezium-embedded-0.10.0.Final.jar:0.10.0.Final]
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[na:na]
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[na:na]
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
    at java.base/java.lang.Thread.run(Thread.java:830) ~[na:na]

The reason is that, org.apache.kafka.connect.storage. KafkaOffsetBackingStore is using bootstrap servers set via Kafka Connect configs, hence we need to see how we can go around this.

Issue Links

• links to: GitHub Pull Request #3530