Kafka
  1. Kafka
  2. KAFKA-808

Migration tool internal queue between consumer and producer threads should be configurable

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: None
    • Component/s: tools
    • Labels:

      Description

      Migration tool maintains an internal blocking queue to buffer messages between the 0.7 consumer and the 0.8 producer. It is fixed in size and it set to the number of producers. However, the consumer usually fetches data in large chunks, several thousands of messages. Also each producer's internal queue frees up in chunks of the producer's batch size. In performance profiling, I saw that the migration tool's consumer thread spent time waiting for space to free up in the internal queue. It seems that this internal queue should be configurable and much larger than the number of producers. This will improve the buffering and help bridge the gap between the consumer and producer's throughput.

      1. kafka-808.patch
        2 kB
        Neha Narkhede

        Activity

        Neha Narkhede created issue -
        Neha Narkhede made changes -
        Field Original Value New Value
        Status Open [ 1 ] In Progress [ 3 ]
        Hide
        Neha Narkhede added a comment -

        Added an optional queue.size config to the migration tool

        Show
        Neha Narkhede added a comment - Added an optional queue.size config to the migration tool
        Neha Narkhede made changes -
        Attachment kafka-808.patch [ 12573737 ]
        Neha Narkhede made changes -
        Status In Progress [ 3 ] Patch Available [ 10002 ]
        Hide
        Jun Rao added a comment -

        Thanks for the patch. +1.

        Show
        Jun Rao added a comment - Thanks for the patch. +1.
        Hide
        Neha Narkhede added a comment -

        This is checked in

        Show
        Neha Narkhede added a comment - This is checked in
        Neha Narkhede made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Neha Narkhede made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Neha Narkhede
            Reporter:
            Neha Narkhede
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development