[KAFKA-10470] zstd decompression with small batches is slow and causes excessive GC

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.5.1
• Fix Version/s: 2.8.0
• Assignee: James Yuzawa
• Reporter: Robert Wagner
• Created: 08/Sep/20 17:01
• Updated: 11/Nov/20 02:12
• Resolved: 11/Nov/20 02:12

Description

Similar to KAFKA-5150 but for zstd instead of LZ4, it appears that a large decompression buffer (128kb) created by zstd-jni per batch is causing a significant performance bottleneck.

The next upcoming version of zstd-jni (1.4.5-7) will have a new constructor for ZstdInputStream that allows the client to pass its own buffer.  A similar fix as PR #2967 could be used to have the  ZstdConstructor use a BufferSupplier to re-use the decompression buffer.

Issue Links

• links to: GitHub Pull Request #9499