[KAFKA-6480] Add config to enforce max fetch size on the broker

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Labels: needs-kip
• Assignee: Unassigned
• Reporter: Jason Gustafson
• Created: 24/Jan/18 19:58
• Updated: 24/Jan/18 19:58

Description

Users are increasingly hitting memory problems due to message format down-conversion. The problem is basically that we have to do the down-conversion in memory. Since the default fetch size is 50Mb, it doesn't take that many fetch requests to cause an OOM. One mitigation is KAFKA-6352. It would also be helpful if the broker had a configuration to restrict the maximum allowed fetch size across all consumers. This would also prevent a malicious client from using this in order to DoS the server.