Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-7859

Tune zlib compression in ORC to account for the encoding strategy

Log workAgile BoardRank to TopRank to BottomVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.14.0
    • File Formats
    • None

    Description

      Currently ORC Zlib is slow because several compression strategies ZLib uses is already done by ORC in itself (dictionary, RLE, bit-packing).

      We need to pick between Z_FILTERED, Z_HUFFMAN_ONLY, Z_RLE, Z_FIXED and Z_DEFAULT_STRATEGY according to column stream type.

      For instance an RLE_V2 stream could a use Z_FILTERED compression without invoking the rest of the strategies.

      The string streams can use Z_FIXED compression strategies and so on.

      The core limitation to stick to retain compatibility with the default decompressor, so that these are automatically backward compatible.

      Attachments

        1. HIVE-7859.3.patch
          134 kB
          Prasanth Jayachandran
        2. HIVE-7859.2.patch
          11 kB
          Gopal Vijayaraghavan
        3. HIVE-7859.1.patch
          11 kB
          Gopal Vijayaraghavan

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            gopalv Gopal Vijayaraghavan Assign to me
            gopalv Gopal Vijayaraghavan
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Issue deployment