[IMPALA-12076] Potential performance improvement using ZSTD's ZSTD_decompressDCtx interface

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 4.3.0
• Fix Version/s: Impala 4.4.0
• Component/s: Backend
• Epic Color: ghx-label-12
• Assignee: Joe McDonnell
• Reporter: Joe McDonnell
• Created: 19/Apr/23 20:25
• Updated: 17/Oct/23 23:48
• Resolved: 17/Oct/23 23:48

Description

In ORC-639, they note that ZSTD's simple interface initializes the context on each call to ZSTD_decompress(). When calling ZSTD_decompress() many times, it is better to allocate the context once and use the ZSTD_decompressDCtx() interface to avoid the repeated initialization.

The ZSTD code mentions that here:

 

/*= Decompression context
 *  When decompressing many times,
 *  it is recommended to allocate a context only once,
 *  and re-use it for each successive compression operation.
 *  This will make workload friendlier for system's memory.
 *  Use one context per thread for parallel execution. */
typedef struct ZSTD_DCtx_s ZSTD_DCtx;

We should investigate using this for decompress.h/.cc's ZstandardDecompressor. We already do that for the streaming decompression mode, but this should also apply to block decompression. Something similar is possible for compression as well.