Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-7250

Adaptive compression buffer size for wide tables in ORC

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.14.0
    • Fix Version/s: 0.14.0
    • Component/s: File Formats
    • Labels:

      Description

      If the input table is wide (in the order of 1000s), ORC compression buffer size overhead becomes significant causing OOM issues. To overcome this issue, buffer size should be adaptively chosen based on the available memory and the number of columns.

        Attachments

        1. HIVE-7250.5.patch
          384 kB
          Prasanth Jayachandran
        2. HIVE-7250.4.patch
          382 kB
          Prasanth Jayachandran
        3. HIVE-7250.3.patch
          372 kB
          Prasanth Jayachandran
        4. HIVE-7250.2.patch
          372 kB
          Prasanth Jayachandran
        5. HIVE-7250.1.patch
          374 kB
          Prasanth Jayachandran

          Issue Links

            Activity

              People

              • Assignee:
                prasanth_j Prasanth Jayachandran
                Reporter:
                prasanth_j Prasanth Jayachandran
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: