Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-11807

Set ORC buffer size in relation to set stripe size

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 1.3.0, 2.0.0
    • File Formats
    • None

    Description

      A customer produced ORC files with very small stripe sizes (10k rows/stripe) by setting a small 64MB stripe size and 256K buffer size for a 54 column table. At that size, each of the streams only get a buffer or two before the stripe size is reached. The current code uses the available memory instead of the stripe size and thus doesn't shrink the buffer size if the JVM has much more memory than the stripe size.

      Attachments

        1. HIVE-11807.patch
          16 kB
          Owen O'Malley
        2. HIVE-11807.patch
          123 kB
          Owen O'Malley
        3. HIVE-11807-branch-1.patch
          123 kB
          Prasanth Jayachandran
        4. HIVE-11807.patch
          95 kB
          Prasanth Jayachandran

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. HIVE-12260 Fix TestColumnStatistics and TestJsonFileDump test failures in master

          • Major - Major loss of function.
          • Closed

          Activity

            People

              omalley Owen O'Malley
              omalley Owen O'Malley
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: