[KAFKA-275] max.message.size is not enforced for compressed messages

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.7
• Component/s: core
• Assignee: Unassigned
• Reporter: Neha Narkhede
• Created: 17/Feb/12 01:56
• Updated: 17/Aug/17 12:02
• Resolved: 17/Aug/17 12:02

Description

The max.message.size check is not performed for compressed messages, but only for each message that forms a compressed message. Due to this, even if the max.message.size is set to 1MB, the producer can technically send n 1MB messages as one compressed message. This can cause memory issues on the server as well as deserialization issues on the consumer. The consumer's fetch size has to be > max.message.size in order to be able to read data. If one message is larger than the fetch.size, the consumer will throw an exception and cannot proceed until the fetch.size is increased.

Due to this bug, even if the fetch.size > max.message.size, the consumer can still get stuck on a message that is larger than max.message.size.

Issue Links

• is part of: KAFKA-273 Occassional GZIP errors on the server while writing compressed data to disk

Sub-Tasks

• Enforce max.message.size on the total message size, not just on payload size: Resolved