Hive
  1. Hive
  2. HIVE-947

Add run length encoding into RCFile's block header

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.5.0
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      When RCFile constructing rows, it needs to get column value's length via calling readVLong(). And this should be avoided for fix length or most fix length columns.
      This also should not influence old rcfile files, which means it should also work correctly on files with previous RCFile format.

        Activity

        Hide
        He Yongqiang added a comment -

        hive-947-2009-11-22.patch implemented the run length encoding. This is only a minor improvement, so no test added.

        Show
        He Yongqiang added a comment - hive-947-2009-11-22.patch implemented the run length encoding. This is only a minor improvement, so no test added.
        Hide
        Namit Jain added a comment -

        Since I am not very familiar with this, let us go over it together sometime

        Show
        Namit Jain added a comment - Since I am not very familiar with this, let us go over it together sometime
        Hide
        Ning Zhang added a comment -

        Yongqiang, can you profile a simple query that we talked the other day and see how much CPU this can save? We should test on columns that are mostly the same length (e.g., type int) and variable lengths (string).

        Show
        Ning Zhang added a comment - Yongqiang, can you profile a simple query that we talked the other day and see how much CPU this can save? We should test on columns that are mostly the same length (e.g., type int) and variable lengths (string).
        Hide
        Namit Jain added a comment -

        got a diff in running test partition_wise_fileformat.q

        Show
        Namit Jain added a comment - got a diff in running test partition_wise_fileformat.q
        Hide
        Namit Jain added a comment -

        Committed. Thanks Yongqiang

        Show
        Namit Jain added a comment - Committed. Thanks Yongqiang

          People

          • Assignee:
            He Yongqiang
            Reporter:
            He Yongqiang
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development