Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-12254

MirrorMaker 2.0 creates destination topic with default configs

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • 2.4.0, 2.5.0, 2.4.1, 2.6.0, 2.5.1, 2.7.0, 2.6.1, 2.8.0
    • 2.8.0
    • None
    • None

    Description

      `MirrorSourceConnector` implements the logic for replicating data, configurations, and other metadata between the source and destination clusters. This includes the tasks below:

      1. `refreshTopicPartitions` for syncing topics / partitions from source to destination.
      2. `syncTopicConfigs` for syncing topic configurations from source to destination.

      A limitation is that `computeAndCreateTopicPartitions` creates topics with default configurations on the destination cluster. A separate async task `syncTopicConfigs` is responsible for syncing the topic configs. Before that sync happens, topic configurations could be out of sync between the two clusters.

      In the worst case, this could lead to data loss eg. when we have a compacted topic being mirrored between clusters which is incorrectly created with the default configuration of `cleanup.policy = delete` on the destination before the configurations are sync'd via `syncTopicConfigs`.

      Here is an example of the divergence:

      Source Topic:

      ```

      Topic: foobar PartitionCount: 1 ReplicationFactor: 1 Configs: cleanup.policy=compact,segment.bytes=1073741824

      ```

      Destination Topic:

      ```

      Topic: A.foobar PartitionCount: 1 ReplicationFactor: 1 Configs: segment.bytes=1073741824

      ```

      A safer approach is to ensure that the right configurations are set on the destination cluster before data is replicated to it.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            dhruvilshah Dhruvil Shah
            dhruvilshah Dhruvil Shah
            Rajini Sivaram Rajini Sivaram
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment