Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-7832

Do ORC dictionary check at a finer level and preserve encoding across stripes

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.14.0
    • Fix Version/s: 0.14.0
    • Component/s: None
    • Labels:
      None

      Description

      Currently ORC dictionary check happens while writing the stripe. Just before writing stripe if ratio of dictionary entries to total non-null rows is greater than threshold then the dictionary is discarded. Also, the decision of using dictionary or not is preserved across stripes. This sometimes leads to costly insertion cost of O(logn) for each stripes when there are too many distinct keys.

        Attachments

        1. HIVE-7832.1.patch
          12 kB
          Prasanth Jayachandran
        2. HIVE-7832.2.patch
          12 kB
          Prasanth Jayachandran
        3. HIVE-7832.3.patch
          13 kB
          Prasanth Jayachandran
        4. HIVE-7832.4.patch
          25 kB
          Prasanth Jayachandran
        5. HIVE-7832.5.patch
          24 kB
          Prasanth Jayachandran
        6. HIVE-7832.6.patch
          23 kB
          Prasanth Jayachandran
        7. HIVE-7832.7.patch
          23 kB
          Prasanth Jayachandran
        8. HIVE-7832.8.patch
          23 kB
          Prasanth Jayachandran
        9. HIVE-7832.9.patch
          23 kB
          Prasanth Jayachandran

          Issue Links

            Activity

              People

              • Assignee:
                prasanth_j Prasanth Jayachandran
                Reporter:
                prasanth_j Prasanth Jayachandran
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: