Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-4248

Implement a memory manager for ORC

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.11.0
    • Labels:
      None

      Description

      With the large default stripe size (256MB) and dynamic partitions, it is quite easy for users to run out of memory when writing ORC files. We probably need a solution that keeps track of the total number of concurrent ORC writers and divides the available heap space between them.

        Attachments

        1. HIVE-4248.D9993.1.patch
          19 kB
          Phabricator
        2. HIVE-4248.D9993.2.patch
          19 kB
          Phabricator
        3. HIVE-4248.D9993.4.patch
          24 kB
          Phabricator

          Activity

            People

            • Assignee:
              owen.omalley Owen O'Malley
              Reporter:
              owen.omalley Owen O'Malley
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: