Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-11807

Set ORC buffer size in relation to set stripe size

Log workAgile BoardRank to TopRank to BottomVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.3.0, 2.0.0
    • Component/s: File Formats
    • Labels:
      None

      Description

      A customer produced ORC files with very small stripe sizes (10k rows/stripe) by setting a small 64MB stripe size and 256K buffer size for a 54 column table. At that size, each of the streams only get a buffer or two before the stripe size is reached. The current code uses the available memory instead of the stripe size and thus doesn't shrink the buffer size if the JVM has much more memory than the stripe size.

        Attachments

        1. HIVE-11807.patch
          95 kB
          Prasanth Jayachandran
        2. HIVE-11807.patch
          123 kB
          Owen O'Malley
        3. HIVE-11807.patch
          16 kB
          Owen O'Malley
        4. HIVE-11807-branch-1.patch
          123 kB
          Prasanth Jayachandran

        Issue Links

          Activity

          $i18n.getText('security.level.explanation', $currentSelection) Viewable by All Users
          Cancel

            People

            • Assignee:
              omalley Owen O'Malley Assign to me
              Reporter:
              omalley Owen O'Malley

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment