Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-527

Compression support does numerous byte copies

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • None
    • None
    • compression
    • None

    Description

      The data path for compressing or decompressing messages is extremely inefficient. We do something like 7 complete copies of the data, often for simple things like adding a 4 byte size to the front. I am not sure how this went by unnoticed.

      This is likely the root cause of the performance issues we saw in doing bulk recompression of data in mirror maker.

      The mismatch between the InputStream and OutputStream interfaces and the Message/MessageSet interfaces which are based on byte buffers is the cause of many of these.

      Attachments

        1. KAFKA-527_2015-03-25_13:26:36.patch
          9 kB
          Guozhang Wang
        2. KAFKA-527_2015-03-25_12:08:00.patch
          27 kB
          Yasuhiro Matsuda
        3. KAFKA-527_2015-03-19_21:32:24.patch
          24 kB
          Yasuhiro Matsuda
        4. KAFKA-527_2015-03-16_15:19:29.patch
          20 kB
          Yasuhiro Matsuda
        5. KAFKA-527.patch
          18 kB
          Yasuhiro Matsuda
        6. KAFKA-527.message-copy.history
          2 kB
          Guozhang Wang
        7. java.hprof.snappy.text
          219 kB
          Jay Kreps
        8. java.hprof.no-compression.txt
          116 kB
          Jay Kreps

        Activity

          People

            yasuhiro.matsuda Yasuhiro Matsuda
            jkreps Jay Kreps
            Votes:
            2 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: