Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-11807

Set ORC buffer size in relation to set stripe size

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.3.0, 2.0.0
    • Component/s: File Formats
    • Labels:
      None

      Description

      A customer produced ORC files with very small stripe sizes (10k rows/stripe) by setting a small 64MB stripe size and 256K buffer size for a 54 column table. At that size, each of the streams only get a buffer or two before the stripe size is reached. The current code uses the available memory instead of the stripe size and thus doesn't shrink the buffer size if the JVM has much more memory than the stripe size.

        Attachments

        1. HIVE-11807.patch
          95 kB
          Prasanth Jayachandran
        2. HIVE-11807-branch-1.patch
          123 kB
          Prasanth Jayachandran
        3. HIVE-11807.patch
          123 kB
          Owen O'Malley
        4. HIVE-11807.patch
          16 kB
          Owen O'Malley

          Issue Links

            Activity

              People

              • Assignee:
                omalley Owen O'Malley
                Reporter:
                omalley Owen O'Malley
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: