Hive
  1. Hive
  2. HIVE-3874

Create a new Optimized Row Columnar file format for Hive

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.11.0
    • Labels:
      None

      Description

      There are several limitations of the current RC File format that I'd like to address by creating a new format:

      • each column value is stored as a binary blob, which means:
        • the entire column value must be read, decompressed, and deserialized
        • the file format can't use smarter type-specific compression
        • push down filters can't be evaluated
      • the start of each row group needs to be found by scanning
      • user metadata can only be added to the file when the file is created
      • the file doesn't store the number of rows per a file or row group
      • there is no mechanism for seeking to a particular row number, which is required for external indexes.
      • there is no mechanism for storing light weight indexes within the file to enable push-down filters to skip entire row groups.
      • the type of the rows aren't stored in the file
      1. HIVE-3874.D8529.4.patch
        745 kB
        Phabricator
      2. HIVE-3874.D8871.1.patch
        12 kB
        Phabricator
      3. HIVE-3874.D8529.3.patch
        741 kB
        Phabricator
      4. HIVE-3874.D8529.2.patch
        740 kB
        Phabricator
      5. HIVE-3874.D8529.1.patch
        735 kB
        Phabricator
      6. hive.3874.2.patch
        666 kB
        Namit Jain
      7. orc.tgz
        49 kB
        Owen O'Malley
      8. OrcFileIntro.pptx
        1.10 MB
        Owen O'Malley

        Issue Links

          Activity

          Owen O'Malley created issue -
          Owen O'Malley made changes -
          Field Original Value New Value
          Attachment OrcFileIntro.pptx [ 12564124 ]
          Owen O'Malley made changes -
          Link This issue blocks HIVE-3889 [ HIVE-3889 ]
          Owen O'Malley made changes -
          Attachment orc.tgz [ 12565524 ]
          Owen O'Malley made changes -
          Attachment orc.tgz [ 12565569 ]
          Owen O'Malley made changes -
          Attachment orc.tgz [ 12565569 ]
          Owen O'Malley made changes -
          Attachment orc.tgz [ 12565524 ]
          Owen O'Malley made changes -
          Attachment orc.tgz [ 12566052 ]
          Namit Jain made changes -
          Attachment hive.3874.2.patch [ 12567179 ]
          Phabricator made changes -
          Attachment HIVE-3874.D8529.1.patch [ 12569090 ]
          Owen O'Malley made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Kevin Wilfong made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Phabricator made changes -
          Attachment HIVE-3874.D8529.2.patch [ 12570091 ]
          Phabricator made changes -
          Attachment HIVE-3874.D8529.3.patch [ 12570332 ]
          Namit Jain made changes -
          Link This issue blocks HIVE-4058 [ HIVE-4058 ]
          Namit Jain made changes -
          Link This issue blocks HIVE-4059 [ HIVE-4059 ]
          Namit Jain made changes -
          Link This issue blocks HIVE-4060 [ HIVE-4060 ]
          Namit Jain made changes -
          Link This issue blocks HIVE-4061 [ HIVE-4061 ]
          Namit Jain made changes -
          Link This issue blocks HIVE-4062 [ HIVE-4062 ]
          Namit Jain made changes -
          Link This issue blocks HIVE-4015 [ HIVE-4015 ]
          Namit Jain made changes -
          Link This issue blocks HIVE-4063 [ HIVE-4063 ]
          Phabricator made changes -
          Attachment HIVE-3874.D8871.1.patch [ 12570810 ]
          Phabricator made changes -
          Attachment HIVE-3874.D8529.4.patch [ 12570813 ]
          Owen O'Malley made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Kevin Wilfong made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Owen O'Malley made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Owen O'Malley made changes -
          Link This issue blocks HIVE-4098 [ HIVE-4098 ]
          Owen O'Malley made changes -
          Link This issue blocks HIVE-4097 [ HIVE-4097 ]
          Kevin Wilfong made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Owen O'Malley made changes -
          Fix Version/s 0.11.0 [ 12323587 ]
          Lefty Leverenz made changes -
          Link This issue is related to HIVE-4376 [ HIVE-4376 ]
          Gavin made changes -
          Link This issue blocks HIVE-3889 [ HIVE-3889 ]
          Gavin made changes -
          Link This issue is depended upon by HIVE-3889 [ HIVE-3889 ]
          Gavin made changes -
          Link This issue blocks HIVE-4060 [ HIVE-4060 ]
          Gavin made changes -
          Link This issue is depended upon by HIVE-4060 [ HIVE-4060 ]
          Gavin made changes -
          Link This issue blocks HIVE-4058 [ HIVE-4058 ]
          Gavin made changes -
          Link This issue is depended upon by HIVE-4058 [ HIVE-4058 ]
          Gavin made changes -
          Link This issue blocks HIVE-4063 [ HIVE-4063 ]
          Gavin made changes -
          Link This issue is depended upon by HIVE-4063 [ HIVE-4063 ]
          Gavin made changes -
          Link This issue blocks HIVE-4059 [ HIVE-4059 ]
          Gavin made changes -
          Link This issue is depended upon by HIVE-4059 [ HIVE-4059 ]
          Gavin made changes -
          Link This issue blocks HIVE-4061 [ HIVE-4061 ]
          Gavin made changes -
          Link This issue is depended upon by HIVE-4061 [ HIVE-4061 ]
          Gavin made changes -
          Link This issue blocks HIVE-4062 [ HIVE-4062 ]
          Gavin made changes -
          Link This issue is depended upon by HIVE-4062 [ HIVE-4062 ]
          Gavin made changes -
          Link This issue blocks HIVE-4015 [ HIVE-4015 ]
          Gavin made changes -
          Link This issue is depended upon by HIVE-4015 [ HIVE-4015 ]
          Gavin made changes -
          Link This issue blocks HIVE-4098 [ HIVE-4098 ]
          Gavin made changes -
          Link This issue is depended upon by HIVE-4098 [ HIVE-4098 ]
          Gavin made changes -
          Link This issue blocks HIVE-4097 [ HIVE-4097 ]
          Gavin made changes -
          Link This issue is depended upon by HIVE-4097 [ HIVE-4097 ]
          Owen O'Malley made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Owen O'Malley
              Reporter:
              Owen O'Malley
            • Votes:
              6 Vote for this issue
              Watchers:
              60 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development