[KAFKA-527] Compression support does numerous byte copies

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Component/s: compression
• Assignee: Yasuhiro Matsuda
• Reporter: Jay Kreps
• Created: 24/Sep/12 20:38
• Updated: 29/Mar/15 22:53
• Resolved: 26/Mar/15 22:51

Description

The data path for compressing or decompressing messages is extremely inefficient. We do something like 7 complete copies of the data, often for simple things like adding a 4 byte size to the front. I am not sure how this went by unnoticed.

This is likely the root cause of the performance issues we saw in doing bulk recompression of data in mirror maker.

The mismatch between the InputStream and OutputStream interfaces and the Message/MessageSet interfaces which are based on byte buffers is the cause of many of these.