Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-7859

Tune zlib compression in ORC to account for the encoding strategy

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.14.0
    • Component/s: File Formats
    • Labels:
      None

      Description

      Currently ORC Zlib is slow because several compression strategies ZLib uses is already done by ORC in itself (dictionary, RLE, bit-packing).

      We need to pick between Z_FILTERED, Z_HUFFMAN_ONLY, Z_RLE, Z_FIXED and Z_DEFAULT_STRATEGY according to column stream type.

      For instance an RLE_V2 stream could a use Z_FILTERED compression without invoking the rest of the strategies.

      The string streams can use Z_FIXED compression strategies and so on.

      The core limitation to stick to retain compatibility with the default decompressor, so that these are automatically backward compatible.

        Attachments

        1. HIVE-7859.3.patch
          134 kB
          Prasanth Jayachandran
        2. HIVE-7859.2.patch
          11 kB
          Gopal V
        3. HIVE-7859.1.patch
          11 kB
          Gopal V

          Activity

            People

            • Assignee:
              gopalv Gopal V
              Reporter:
              gopalv Gopal V
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: