Kafka
  1. Kafka
  2. KAFKA-844

System Test - Mirror Maker cases enhancements

    Details

    • Type: Task Task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      The following changes are needed to enhance System Test Mirror Maker cases:
      1. Auto create topic
      2. Specify num.partitions = 1
      3. Add ConsoleConsumer to consume data from source cluster for validation reference

      1. kafka-844-v3.patch
        20 kB
        John Fung
      2. kafka-844-v2.patch
        20 kB
        John Fung
      3. kafka-844-v1.patch
        17 kB
        John Fung

        Activity

        John Fung created issue -
        John Fung made changes -
        Field Original Value New Value
        Attachment kafka-844-v1.patch [ 12576597 ]
        John Fung made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Neha Narkhede added a comment -

        Thanks for the patch, John! I think changes 1and 4 make sense. However, I don't think we should make change #2. This might hide some issues that show up with multiple partitions. Let's set the number of partitions to at least 2 or 3 and bump up producer retries if required. Also with the fix for KAFKA-842, I don't think you need change #3 as well, let's confirm that.

        Show
        Neha Narkhede added a comment - Thanks for the patch, John! I think changes 1and 4 make sense. However, I don't think we should make change #2. This might hide some issues that show up with multiple partitions. Let's set the number of partitions to at least 2 or 3 and bump up producer retries if required. Also with the fix for KAFKA-842 , I don't think you need change #3 as well, let's confirm that.
        John Fung made changes -
        Attachment kafka-844-v2.patch [ 12576644 ]
        Hide
        John Fung added a comment - - edited

        Thanks Neha for reviewing. The num.partitions is set to 5 and this patch is assumed to be applied on top of KAFKA-842. (original #3 (wait longer after mirror maker is terminated) is removed.

        Show
        John Fung added a comment - - edited Thanks Neha for reviewing. The num.partitions is set to 5 and this patch is assumed to be applied on top of KAFKA-842 . (original #3 (wait longer after mirror maker is terminated) is removed.
        John Fung made changes -
        Description The following changes are needed to enhance System Test Mirror Maker cases:
        1. Auto create topic
        2. Specify num.partitions = 1
        3. Wait longer after mirror maker is terminated by kill -15
        4. Add ConsoleConsumer to consume data from source cluster for validation reference
        The following changes are needed to enhance System Test Mirror Maker cases:
        1. Auto create topic
        2. Specify num.partitions = 1
        3. Add ConsoleConsumer to consume data from source cluster for validation reference
        John Fung made changes -
        Attachment kafka-844-v2.patch [ 12576644 ]
        John Fung made changes -
        Attachment kafka-844-v2.patch [ 12576670 ]
        Hide
        Neha Narkhede added a comment -

        Thanks for the patch! Just one question -

        I saw that you set producer retries to 25. This seems very high. What is the lowest value for producer retries that works for 5 partitions and 5 producer threads ?

        Show
        Neha Narkhede added a comment - Thanks for the patch! Just one question - I saw that you set producer retries to 25. This seems very high. What is the lowest value for producer retries that works for 5 partitions and 5 producer threads ?
        Hide
        John Fung added a comment -

        Thanks Neha for reviewing the patch. The Mirror Maker test cases are working fine with producer retries set to 3.

        Uploaded kafka-844-v3.patch to have message.send.max.retries=3

        Show
        John Fung added a comment - Thanks Neha for reviewing the patch. The Mirror Maker test cases are working fine with producer retries set to 3. Uploaded kafka-844-v3.patch to have message.send.max.retries=3
        John Fung made changes -
        Attachment kafka-844-v3.patch [ 12577018 ]
        Hide
        Neha Narkhede added a comment -

        +1 on patch v3

        Show
        Neha Narkhede added a comment - +1 on patch v3
        Hide
        Neha Narkhede added a comment -

        Committed v3 to 0.8

        Show
        Neha Narkhede added a comment - Committed v3 to 0.8
        Neha Narkhede made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Neha Narkhede made changes -
        Assignee John Fung [ jfung ]
        Neha Narkhede made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            John Fung
            Reporter:
            John Fung
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development