Kafka
  1. Kafka
  2. KAFKA-567

Replication Data Loss in Mirror Maker Bouncing testcase

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: 0.8.0
    • Component/s: core
    • Labels:

      Description

      • Test Description:
        1. Start a 3-broker cluster as source
        2. Start a 3-broker cluster as target
        3. Start 1 instance of Mirror Maker to replicate data from source to target
        4. While producer is sending data into source cluster, stop Mirror Maker with "kill -15". Start Mirror Maker again after 1 second.
        5. Start a consumer to consume data from target cluster.
        6. Compare the MessageID in the data between producer log and consumer log.
      • To reproduce this issue, please do the followings:

      1. Download the latest 0.8 branch
      2. Apply the patch attached to this JIRA
      3. Build kafka by running "./sbt update package"
      4. Execute the test in directory "system_test" : "python -B system_test_runner.py"

      • The test result may look like the following:

      _test_case_name : testcase_5002
      _test_class_name : MirrorMakerTest
      arg : bounce_leader : false
      arg : bounce_mirror_maker : true
      arg : message_producing_free_time_sec : 15
      arg : num_iteration : 1
      arg : num_messages_to_produce_per_producer_call : 50
      arg : num_partition : 1
      arg : replica_factor : 3
      arg : sleep_seconds_between_producer_calls : 1
      validation_status :
      Log segment checksum matching across all replicas : FAILED
      Unique messages from consumer on [test_1] : 355
      Unique messages from producer on [test_1] : 400
      Validate for data matched on topic [test_1] : FAILED

      • Attached a tar file for the system test output log, the brokers' log4j files and data log segment files.
      • There are no unusual Exception / Error found in the logs. However, there are consistently data loss in this Mirror Maker bouncing test case. Not sure if this is related to KAFKA-552.
      1. system_test_1349971807_testcase_5002.tar
        2.01 MB
        John Fung
      2. mirror_maker_12.log
        219 kB
        John Fung
      3. kafka-mirror-maker-bouncing-data-loss.patch
        18 kB
        John Fung
      4. KAFKA-567-mirrormakershutdown-v1.patch
        2 kB
        Joel Koshy
      5. kafka-567.patch
        0.6 kB
        Jun Rao

        Activity

        John Fung created issue -
        John Fung made changes -
        Field Original Value New Value
        Attachment kafka-mirror-maker-bouncing-data-loss.patch [ 12548773 ]
        John Fung made changes -
        John Fung made changes -
        Description * To reproduce this issue, please do the followings:

        1. Download the latest 0.8 branch
        2. Apply the patch attached to this JIRA
        3. Build kafka by running "./sbt update package"
        4. Execute the test in directory "system_test" : "python -B system_test_runner.py"

        * The test result may look like the following:

        _test_case_name : testcase_5002
        _test_class_name : MirrorMakerTest
        arg : bounce_leader : false
        arg : bounce_mirror_maker : true
        arg : message_producing_free_time_sec : 15
        arg : num_iteration : 1
        arg : num_messages_to_produce_per_producer_call : 50
        arg : num_partition : 1
        arg : replica_factor : 3
        arg : sleep_seconds_between_producer_calls : 1
        validation_status :
             Log segment checksum matching across all replicas : FAILED
             Unique messages from consumer on [test_1] : 355
             Unique messages from producer on [test_1] : 400
             Validate for data matched on topic [test_1] : FAILED

        * Attached a tar file for the system test output log, the brokers' log4j files and data log segment files.

        * There are no unusual Exception / Error found in the logs. However, there are consistently data loss in this Mirror Maker bouncing test case. Not sure if this is related to KAFKA-552.
        * Test Description:
        1. Start a 3-broker cluster as source
        2. Start a 3-broker cluster as target
        3. Start 1 instance of Mirror Maker to replicate data from source to target
        4. While producer is sending data into source cluster, stop Mirror Maker with "kill -15". Start Mirror Maker again after 1 second.
        5. Start a consumer to consume data from target cluster.
        6. Compare the MessageID in the data between producer log and consumer log.

        * To reproduce this issue, please do the followings:

        1. Download the latest 0.8 branch
        2. Apply the patch attached to this JIRA
        3. Build kafka by running "./sbt update package"
        4. Execute the test in directory "system_test" : "python -B system_test_runner.py"

        * The test result may look like the following:

        _test_case_name : testcase_5002
        _test_class_name : MirrorMakerTest
        arg : bounce_leader : false
        arg : bounce_mirror_maker : true
        arg : message_producing_free_time_sec : 15
        arg : num_iteration : 1
        arg : num_messages_to_produce_per_producer_call : 50
        arg : num_partition : 1
        arg : replica_factor : 3
        arg : sleep_seconds_between_producer_calls : 1
        validation_status :
             Log segment checksum matching across all replicas : FAILED
             Unique messages from consumer on [test_1] : 355
             Unique messages from producer on [test_1] : 400
             Validate for data matched on topic [test_1] : FAILED

        * Attached a tar file for the system test output log, the brokers' log4j files and data log segment files.

        * There are no unusual Exception / Error found in the logs. However, there are consistently data loss in this Mirror Maker bouncing test case. Not sure if this is related to KAFKA-552.
        John Fung made changes -
        Attachment mirror_maker_12.log [ 12548793 ]
        Jun Rao made changes -
        Attachment kafka-567.patch [ 12548853 ]
        Jun Rao made changes -
        Assignee Jun Rao [ junrao ]
        Jun Rao made changes -
        Affects Version/s 0.8 [ 12317244 ]
        Labels bugs
        Priority Major [ 3 ] Blocker [ 1 ]
        Component/s core [ 12315217 ]
        Joel Koshy made changes -
        Attachment KAFKA-567-mirrormakershutdown-v1.patch [ 12548913 ]
        Jun Rao made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Jun Rao made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Fix Version/s 0.8 [ 12317244 ]
        Resolution Fixed [ 1 ]
        Jun Rao made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Swapnil Ghike made changes -
        Comment [ Thanks for kafka-567.patch! I was also hitting this bug with the container change. ]

          People

          • Assignee:
            Jun Rao
            Reporter:
            John Fung
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development