Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-13796

MM2 - Topics Exclude/Blacklist not working

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Blocker
    • Resolution: Unresolved
    • 3.0.0
    • 3.0.0
    • mirrormaker
    • None

    Description

      Hi all,

      We are testing Kafka 3.0.0 in a development environment with Java 17 in a Centos 7 machine.

      When setting up MM2 replication between two clusters we exclude some topics we don't want MM2 to replicate via topics.exclude (we have also tried topics.blacklist)

      Even though the configuration is parsed correctly, we still get those topics replicated among clusters.

      Here is our config for the us_central cluster:

      ```

          # Clusters and bootstrap servers
          clusters=eu_west, us_central
          eu_west.bootstrap.servers=XXnet:9092,XX:9092
          us_central.bootstrap.servers=XX:9092

          us_central.consumer.auto.offset.reset=latest
          us_central.consumer.fetch.max.bytes=31457280
          us_central.consumer.fetch.max.wait.ms=10000
          us_central.consumer.max.poll.records=3000
          us_central.consumer.request.timeout.ms=60000

          eu_west.consumer.auto.offset.reset=latest
          eu_west.consumer.fetch.max.bytes=52428800
          eu_west.consumer.fetch.max.wait.ms=10000
          eu_west.consumer.max.poll.records=3000
          eu_west.consumer.request.timeout.ms=60000

          # Custom producer settings
          us_central.producer.max.request.size=27262976
          us_central.producer.batch.size=22000
          us_central.producer.compression.type=none
          us_central.producer.send.buffer.bytes=26214400
          us_central.producer.receive.buffer.bytes=26214400

          eu_west.producer.max.request.size=27262976
          eu_west.producer.batch.size=22000
          eu_west.producer.compression.type=none
          eu_west.producer.send.buffer.bytes=26214400
          eu_west.producer.receive.buffer.bytes=26214400

          # Topics configuration
          topics=.*
          topics.blacklist="mm2_test_0"

          # Tasks (threads)
          tasks.max=10

          # Groups
          groups=phx_netflow

          # Replication factors, these are 1 for lab
          replication.factor=1
          config.storage.replication.factor=1
          offset.storage.replication.factor=1
          status.storage.replication.factor=1
          checkpoints.topic.replication.factor=1
          offset-syncs.topic.replication.factor=1
          sync.group.offsets.replication.factor=1

          ## Config
          config.properties.exclude=local.retention.ms

          # Refresh rates
          us_central.refresh.topics.interval.seconds=15
          us_central.refresh.groups.interval.seconds=15

          eu_west.refresh.topics.interval.seconds=15
          eu_west.refresh.groups.interval.seconds=15

          ## Sync options
          sync.topic.acls.enabled=true
          offset-syncs.topic.replication.enabled=true
          sync.group.offsets.enabled=true

          # EU pulls us, syd, nyc
          us_central->eu_west.enabled=true
          us_central->eu_west.sync.group.offsets.enabled=false

          # US pulls eu, syd, nyc
          eu_west->us_central.enabled=true
          eu_west->us_central.sync.group.offsets.enabled=true

      ```

      Here is the configuration for eu_west cluster:

      ```

          # Clusters and bootstrap servers
          clusters=eu_west, us_central
          eu_west.bootstrap.servers=XX,XX:9092
          us_central.bootstrap.servers=XX:9092

          us_central.consumer.auto.offset.reset=latest
          us_central.consumer.fetch.max.bytes=31457280
          us_central.consumer.fetch.max.wait.ms=10000
          us_central.consumer.max.poll.records=3000
          us_central.consumer.request.timeout.ms=60000

          eu_west.consumer.auto.offset.reset=latest
          eu_west.consumer.fetch.max.bytes=52428800
          eu_west.consumer.fetch.max.wait.ms=10000
          eu_west.consumer.max.poll.records=3000
          eu_west.consumer.request.timeout.ms=60000

          # Custom producer settings
          us_central.producer.max.request.size=27262976
          us_central.producer.batch.size=22000
          us_central.producer.compression.type=none
          us_central.producer.send.buffer.bytes=26214400
          us_central.producer.receive.buffer.bytes=26214400

          eu_west.producer.max.request.size=27262976
          eu_west.producer.batch.size=22000
          eu_west.producer.compression.type=none
          eu_west.producer.send.buffer.bytes=26214400
          eu_west.producer.receive.buffer.bytes=26214400

          # Topics configuration
          topics=.*
          topics.blacklist="mm2_test_0"

          # Tasks (threads)
          tasks.max=10

          # Groups
          groups=ams_netflow

          # Replication factors, these are 1 for lab
          replication.factor=1
          config.storage.replication.factor=1
          offset.storage.replication.factor=1
          status.storage.replication.factor=1
          checkpoints.topic.replication.factor=1
          offset-syncs.topic.replication.factor=1
          sync.group.offsets.replication.factor=1

          ## Config
         config.properties.exclude=local.retention.ms

          # Refresh rates
          us_central.refresh.topics.interval.seconds=15
          us_central.refresh.groups.interval.seconds=15

          eu_west.refresh.topics.interval.seconds=15
          eu_west.refresh.groups.interval.seconds=15

          ## Sync options
          sync.topic.acls.enabled=true
          offset-syncs.topic.replication.enabled=true
          sync.group.offsets.enabled=true

          # EU pulls us, syd, nyc
          us_central->eu_west.enabled=true
          us_central->eu_west.sync.group.offsets.enabled=true

          # US pulls eu, syd, nyc
          eu_west->us_central.enabled=true
          eu_west->us_central.sync.group.offsets.enabled=false

      ```

      When starting MM2 freshly installed (deleted all mm2 related topics), we still get the excluded topics replicated between clusters.

       It's also worth mentioning that the connector seems to be reading the configuration correctly however it is (probably) at a later stage where the topics get filtered incorrectly.

      I attach a log from our connect.log where the exclude field is correctly parsed from our connect-mirrormaker.properties.

      ```

          groups.exclude = [console-consumer-.*, connect-.*, __.*]
          topics.exclude = [mm2_test_0]
          exclude.internal.topics = true

      ```

      After much testing and over 20 combinations of regex we have not found a way to exclude the topics via the configuration properties specified in the documentation for MM2.

      The whitelist does work though, so we are a bit confused in why the blacklist does not.

      We request help to further test this, we are not sure if we are doing something wrong.

       

      Thanks

      Attachments

        Activity

          People

            Unassigned Unassigned
            dbros David Bros
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated: