Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-17188

ObjectStore runs out of memory for large batches of addPartitions().

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.2.0
    • Fix Version/s: 2.2.0, 3.0.0, 2.4.0
    • Component/s: Metastore
    • Labels:
      None

      Description

      For large batches (e.g. hundreds) of addPartitions(), the ObjectStore runs out of memory. Flushing the PersistenceManager alleviates the problem.

      Note: The problem being addressed here isn't so much with the size of the hundreds of Partition objects, but the cruft that builds with the PersistenceManager, in the JDO layer, as confirmed through memory-profiling.

      (Raising this on behalf of Chris Drome and Thiruvel Thirumoolan.)

        Attachments

        1. HIVE-17188.1.patch
          0.8 kB
          Mithun Radhakrishnan

          Activity

            People

            • Assignee:
              cdrome Chris Drome
              Reporter:
              mithun Mithun Radhakrishnan
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: