Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-527

Compression support does numerous byte copies

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: compression
    • Labels:
      None

      Description

      The data path for compressing or decompressing messages is extremely inefficient. We do something like 7 complete copies of the data, often for simple things like adding a 4 byte size to the front. I am not sure how this went by unnoticed.

      This is likely the root cause of the performance issues we saw in doing bulk recompression of data in mirror maker.

      The mismatch between the InputStream and OutputStream interfaces and the Message/MessageSet interfaces which are based on byte buffers is the cause of many of these.

        Attachments

        1. java.hprof.no-compression.txt
          116 kB
          Jay Kreps
        2. java.hprof.snappy.text
          219 kB
          Jay Kreps
        3. KAFKA-527.message-copy.history
          2 kB
          Guozhang Wang
        4. KAFKA-527.patch
          18 kB
          Yasuhiro Matsuda
        5. KAFKA-527_2015-03-16_15:19:29.patch
          20 kB
          Yasuhiro Matsuda
        6. KAFKA-527_2015-03-19_21:32:24.patch
          24 kB
          Yasuhiro Matsuda
        7. KAFKA-527_2015-03-25_12:08:00.patch
          27 kB
          Yasuhiro Matsuda
        8. KAFKA-527_2015-03-25_13:26:36.patch
          9 kB
          Guozhang Wang

          Activity

            People

            • Assignee:
              yasuhiro.matsuda Yasuhiro Matsuda
              Reporter:
              jkreps Jay Kreps
            • Votes:
              2 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: