[KAFKA-17296] Add a new compression option on the broker side

• Type: New Feature
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 3.8.0
• Fix Version/s: 4.0.0
• Component/s: compression, config
• Labels: kip
• Assignee: Unassigned
• Reporter: xiaochen.zhou
• Created: 08/Aug/24 04:24
• Updated: 01/Sep/24 14:07

Description

Compressing data can reduce the bandwidth required for network transmission and decrease disk storage usage. If the compression types of the client and broker are inconsistent, it may lead to increased resource usage on the broker, such as higher CPU usage. Currently, the default compression type on the Kafka broker side is "PRODUCER". As I understand it, if the client does not use compression, the broker will not use compression either.

Can we add a new compression option on the broker side, such as "PRODUCER_OR_LZ4". With this option, if the client uses a compression type (like Snappy), the broker would follow the client's compression type (Snappy). If the client does not use compression, the broker would default to using LZ4.

If this suggestion makes sense, I would be happy to implement it.