Uploaded image for project: 'Kudu'
  1. Kudu
  2. KUDU-1600

Disallow enabling compression on already-compressed encodings

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.3.0
    • Component/s: cfile, master, perf
    • Labels:
      None

      Description

      Some encodings (eg bitshuffle) are already compressed with LZ4, and enabling another pass of LZ4 compression on top of it just burns CPU. We should disallow such pairs of encodings.

      This is slightly trickier in the case of something like dictionary encoding, since it has the fallback path. For blocks that have fallen back to plain encoding, LZ4 is a good idea, whereas for blocks that are dict-encoded, the code words are already compressed and LZ4 is a waste.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                tlipcon Todd Lipcon
                Reporter:
                tlipcon Todd Lipcon
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: