[KAFKA-3160] Kafka LZ4 framing code miscalculates header checksum

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.8.2.0, 0.8.2.1, 0.8.2.2, 0.9.0.0, 0.9.0.1
• Fix Version/s: 0.10.0.0
• Component/s: compression
• Labels: compatibility compression lz4
• Assignee: Dana Powers
• Reporter: Dana Powers
• Created: 27/Jan/16 19:43
• Updated: 09/May/16 17:52
• Resolved: 07/May/16 18:36

Description

KAFKA-1493 partially implements the LZ4 framing specification, but it incorrectly calculates the header checksum. This causes KafkaLZ4BlockInputStream to raise an error [IOException(DESCRIPTOR_HASH_MISMATCH)] if a client sends correctly framed LZ4 data. It also causes KafkaLZ4BlockOutputStream to generate incorrectly framed LZ4 data, which means clients decoding LZ4 messages from kafka will always receive incorrectly framed data.

Specifically, the current implementation includes the 4-byte MagicNumber in the checksum, which is incorrect.
http://cyan4973.github.io/lz4/lz4_Frame_format.html

Third-party clients that attempt to use off-the-shelf lz4 framing find that brokers reject messages as having a corrupt checksum. So currently non-java clients must 'fixup' lz4 packets to deal with the broken checksum.

Magnus first identified this issue in librdkafka; kafka-python has the same problem.