Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-4005

Column truncation

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.11.0
    • Fix Version/s: 0.12.0
    • Component/s: CLI
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Column truncation allows users to remove data for columns that are no longer useful.

      This is done by removing the data for the column and setting the length of the column data and related lengths to 0 in the RC file header.

      RC file was fixed to recognize columns with lengths of zero to be empty and are treated as if the column doesn't exist in the data, a null is returned for every value of that column in every row. This is the same thing that happens when more columns are selected than exist in the file.

      A new command was added to the CLI
      TRUNCATE TABLE ... PARTITION ... COLUMNS ...

      This launches a map only job where each mapper rewrites a single file without the unnecessary column data and the adjusted headers. It does not uncompress/deserialize the data so it is much faster than rewriting the data with NULLs.

        Attachments

        1. HIVE-4005.1.patch.txt
          107 kB
          Kevin Wilfong
        2. HIVE-4005.2.patch.txt
          105 kB
          Kevin Wilfong
        3. HIVE-4005.3.patch.txt
          125 kB
          Kevin Wilfong
        4. HIVE-4005.4.patch.txt
          129 kB
          Kevin Wilfong
        5. HIVE-4005.5.patch.txt
          129 kB
          Kevin Wilfong
        6. HIVE-4005.6.patch.txt
          130 kB
          Kevin Wilfong
        7. HIVE-4005.6.patch.txt
          129 kB
          Kevin Wilfong
        8. HIVE-4005.7.patch.txt
          130 kB
          Kevin Wilfong

          Activity

            People

            • Assignee:
              kevinwilfong Kevin Wilfong
              Reporter:
              kevinwilfong Kevin Wilfong
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: