Details
-
Bug
-
Status: Open
-
Blocker
-
Resolution: Unresolved
-
3.0.0
-
None
Description
Hi all,
We are testing Kafka 3.0.0 in a development environment with Java 17 in a Centos 7 machine.
When setting up MM2 replication between two clusters we exclude some topics we don't want MM2 to replicate via topics.exclude (we have also tried topics.blacklist)
Even though the configuration is parsed correctly, we still get those topics replicated among clusters.
Here is our config for the us_central cluster:
```
# Clusters and bootstrap servers
clusters=eu_west, us_central
eu_west.bootstrap.servers=XXnet:9092,XX:9092
us_central.bootstrap.servers=XX:9092
us_central.consumer.auto.offset.reset=latest
us_central.consumer.fetch.max.bytes=31457280
us_central.consumer.fetch.max.wait.ms=10000
us_central.consumer.max.poll.records=3000
us_central.consumer.request.timeout.ms=60000
eu_west.consumer.auto.offset.reset=latest
eu_west.consumer.fetch.max.bytes=52428800
eu_west.consumer.fetch.max.wait.ms=10000
eu_west.consumer.max.poll.records=3000
eu_west.consumer.request.timeout.ms=60000
# Custom producer settings
us_central.producer.max.request.size=27262976
us_central.producer.batch.size=22000
us_central.producer.compression.type=none
us_central.producer.send.buffer.bytes=26214400
us_central.producer.receive.buffer.bytes=26214400
eu_west.producer.max.request.size=27262976
eu_west.producer.batch.size=22000
eu_west.producer.compression.type=none
eu_west.producer.send.buffer.bytes=26214400
eu_west.producer.receive.buffer.bytes=26214400
# Topics configuration
topics=.*
topics.blacklist="mm2_test_0"
# Tasks (threads)
tasks.max=10
# Groups
groups=phx_netflow
# Replication factors, these are 1 for lab
replication.factor=1
config.storage.replication.factor=1
offset.storage.replication.factor=1
status.storage.replication.factor=1
checkpoints.topic.replication.factor=1
offset-syncs.topic.replication.factor=1
sync.group.offsets.replication.factor=1
## Config
config.properties.exclude=local.retention.ms
# Refresh rates
us_central.refresh.topics.interval.seconds=15
us_central.refresh.groups.interval.seconds=15
eu_west.refresh.topics.interval.seconds=15
eu_west.refresh.groups.interval.seconds=15
## Sync options
sync.topic.acls.enabled=true
offset-syncs.topic.replication.enabled=true
sync.group.offsets.enabled=true
# EU pulls us, syd, nyc
us_central->eu_west.enabled=true
us_central->eu_west.sync.group.offsets.enabled=false
# US pulls eu, syd, nyc
eu_west->us_central.enabled=true
eu_west->us_central.sync.group.offsets.enabled=true
```
Here is the configuration for eu_west cluster:
```
# Clusters and bootstrap servers
clusters=eu_west, us_central
eu_west.bootstrap.servers=XX,XX:9092
us_central.bootstrap.servers=XX:9092
us_central.consumer.auto.offset.reset=latest
us_central.consumer.fetch.max.bytes=31457280
us_central.consumer.fetch.max.wait.ms=10000
us_central.consumer.max.poll.records=3000
us_central.consumer.request.timeout.ms=60000
eu_west.consumer.auto.offset.reset=latest
eu_west.consumer.fetch.max.bytes=52428800
eu_west.consumer.fetch.max.wait.ms=10000
eu_west.consumer.max.poll.records=3000
eu_west.consumer.request.timeout.ms=60000
# Custom producer settings
us_central.producer.max.request.size=27262976
us_central.producer.batch.size=22000
us_central.producer.compression.type=none
us_central.producer.send.buffer.bytes=26214400
us_central.producer.receive.buffer.bytes=26214400
eu_west.producer.max.request.size=27262976
eu_west.producer.batch.size=22000
eu_west.producer.compression.type=none
eu_west.producer.send.buffer.bytes=26214400
eu_west.producer.receive.buffer.bytes=26214400
# Topics configuration
topics=.*
topics.blacklist="mm2_test_0"
# Tasks (threads)
tasks.max=10
# Groups
groups=ams_netflow
# Replication factors, these are 1 for lab
replication.factor=1
config.storage.replication.factor=1
offset.storage.replication.factor=1
status.storage.replication.factor=1
checkpoints.topic.replication.factor=1
offset-syncs.topic.replication.factor=1
sync.group.offsets.replication.factor=1
## Config
config.properties.exclude=local.retention.ms
# Refresh rates
us_central.refresh.topics.interval.seconds=15
us_central.refresh.groups.interval.seconds=15
eu_west.refresh.topics.interval.seconds=15
eu_west.refresh.groups.interval.seconds=15
## Sync options
sync.topic.acls.enabled=true
offset-syncs.topic.replication.enabled=true
sync.group.offsets.enabled=true
# EU pulls us, syd, nyc
us_central->eu_west.enabled=true
us_central->eu_west.sync.group.offsets.enabled=true
# US pulls eu, syd, nyc
eu_west->us_central.enabled=true
eu_west->us_central.sync.group.offsets.enabled=false
```
When starting MM2 freshly installed (deleted all mm2 related topics), we still get the excluded topics replicated between clusters.
It's also worth mentioning that the connector seems to be reading the configuration correctly however it is (probably) at a later stage where the topics get filtered incorrectly.
I attach a log from our connect.log where the exclude field is correctly parsed from our connect-mirrormaker.properties.
```
groups.exclude = [console-consumer-.*, connect-.*, __.*]
topics.exclude = [mm2_test_0]
exclude.internal.topics = true
```
After much testing and over 20 combinations of regex we have not found a way to exclude the topics via the configuration properties specified in the documentation for MM2.
The whitelist does work though, so we are a bit confused in why the blacklist does not.
We request help to further test this, we are not sure if we are doing something wrong.
Thanks