Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-9616

[DOC]: Document spill to disk options

VotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Documentation
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: Impala 2.7.0
    • Fix Version/s: None
    • Component/s: Docs
    • Labels:
    • Docs Text:
      Hide
      Something like: Enabled via --disk_spill_compression_codec, which uses
      the same syntax as the compression_codec query option.
      Recommended codecs are LZ4 and ZSTD. ZSTD supports
      specifying a compression level.

      To prevent internal fragmentation, a new technique similar to data caching
      frees space management.

      --disk_spill_punch_holes=true. TmpFileMgr will allocate
      a range of the actual compressed size and punch holes
      in the file for each range that is no longer needed.

      UncompressedWriteIoBytes is added to the buffer pool
      profiles, so that you can see what degree of compression
      is achieved. One user reported ratios of 2-3x for LZ4 and
      ZSTD (with LZ4 toward the lower end and ZSTD toward
      the higher end).
      Show
      Something like: Enabled via --disk_spill_compression_codec, which uses the same syntax as the compression_codec query option. Recommended codecs are LZ4 and ZSTD. ZSTD supports specifying a compression level. To prevent internal fragmentation, a new technique similar to data caching frees space management. --disk_spill_punch_holes=true. TmpFileMgr will allocate a range of the actual compressed size and punch holes in the file for each range that is no longer needed. UncompressedWriteIoBytes is added to the buffer pool profiles, so that you can see what degree of compression is achieved. One user reported ratios of 2-3x for LZ4 and ZSTD (with LZ4 toward the lower end and ZSTD toward the higher end).

      Description

      Document that users can optionally compress spilled data.

        Attachments

          Activity

            People

            • Assignee:
              krishahn Kris Hahn
              Reporter:
              krishahn Kris Hahn

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment