[IMPALA-9616] [DOC]: Document spill to disk options - ASF JIRA

XML

Word

Printable

JSON

Docs Text:

Hide
Something like: Enabled via --disk_spill_compression_codec, which uses
the same syntax as the compression_codec query option.
Recommended codecs are LZ4 and ZSTD. ZSTD supports
specifying a compression level.

To prevent internal fragmentation, a new technique similar to data caching
frees space management.

--disk_spill_punch_holes=true. TmpFileMgr will allocate
a range of the actual compressed size and punch holes
in the file for each range that is no longer needed.

UncompressedWriteIoBytes is added to the buffer pool
profiles, so that you can see what degree of compression
is achieved. One user reported ratios of 2-3x for LZ4 and
ZSTD (with LZ4 toward the lower end and ZSTD toward
the higher end).

Show
Something like: Enabled via --disk_spill_compression_codec, which uses the same syntax as the compression_codec query option. Recommended codecs are LZ4 and ZSTD. ZSTD supports specifying a compression level. To prevent internal fragmentation, a new technique similar to data caching frees space management. --disk_spill_punch_holes=true. TmpFileMgr will allocate a range of the actual compressed size and punch holes in the file for each range that is no longer needed. UncompressedWriteIoBytes is added to the buffer pool profiles, so that you can see what degree of compression is achieved. One user reported ratios of 2-3x for LZ4 and ZSTD (with LZ4 toward the lower end and ZSTD toward the higher end).

Document that users can optionally compress spilled data.