Kafka
  1. Kafka
  2. KAFKA-315

enable shallow iterator in ByteBufferMessageSet to allow mirroing data without decompression

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.7.1
    • Component/s: core
    • Labels:
      None

      Description

      Currently, the iterator of ByteBufferMessageSet does deep iteration, ie, if messages are compressed, they will be decompressed first during the iteration. This adds CPU overhead. For mirroring data between 2 Kafka clusters, we can use a shallow iterator to avoid the decompression overhead.

        Activity

        Hide
        Jun Rao added a comment -

        patch attached.

        Show
        Jun Rao added a comment - patch attached.
        Hide
        Joel Koshy added a comment -

        Patch looks good - would be great if you can provide some idea on the performance overhead of decompression vs. compression from the analysis that you did.

        Also, we can add this config under "Important configuration parameters for a mirror" in the mirroring wiki.

        One question: ConsumerIterator still decodes the event - which would generally only make sense if you are using the DefaultDecoder right? So maybe we can just add a condition there to just return item.message if enableShallowIterator is true and also require that decoder is an instance of DefaultDecoder?

        Show
        Joel Koshy added a comment - Patch looks good - would be great if you can provide some idea on the performance overhead of decompression vs. compression from the analysis that you did. Also, we can add this config under "Important configuration parameters for a mirror" in the mirroring wiki. One question: ConsumerIterator still decodes the event - which would generally only make sense if you are using the DefaultDecoder right? So maybe we can just add a condition there to just return item.message if enableShallowIterator is true and also require that decoder is an instance of DefaultDecoder?
        Hide
        Jun Rao added a comment -

        Thanks for the review. Committed to trunk.

        For performance, with a socket buffer and a fetch size of 2MB, I was able to improve the cross DC mirroring throughput btw 2 brokers from about 9MB/sec to 30MB/sec, using this patch.

        As for the decoder, shallowIterator only works with the DefaultDecoder. If a wrong decoder is used, the user will get an exception and realize the decoder problem. So, this is probably not a big concern.

        Show
        Jun Rao added a comment - Thanks for the review. Committed to trunk. For performance, with a socket buffer and a fetch size of 2MB, I was able to improve the cross DC mirroring throughput btw 2 brokers from about 9MB/sec to 30MB/sec, using this patch. As for the decoder, shallowIterator only works with the DefaultDecoder. If a wrong decoder is used, the user will get an exception and realize the decoder problem. So, this is probably not a big concern.

          People

          • Assignee:
            Jun Rao
            Reporter:
            Jun Rao
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development