Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-9359

Export of a large table causes OOM in Metastore and Client

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.0.0
    • Component/s: Import/Export, Metastore
    • Labels:
      None

      Description

      Running hive export on a table with a large number of partitions winds up making the metastore and client run out of memory. The number of places we wind up having a copy of the entire partitions object wind up being as follows:

      Metastore

      • (temporarily) Metastore MPartition objects
      • List<Partition> that gets persisted before sending to thrift
      • thrift copy of all of those partitions

      Client side

      • thrift copy of partitions
      • deepcopy of above to create List<Partition> objects
      • JSONObject that contains all of those above partition objects
      • List<ReadEntity> which each encapsulates the aforesaid partition objects.

      This memory usage needs to be drastically reduced.

        Attachments

        1. HIVE-9359.2.patch
          11 kB
          Sushanth Sowmyan
        2. HIVE-9359.patch
          11 kB
          Sushanth Sowmyan

          Activity

            People

            • Assignee:
              sushanth Sushanth Sowmyan
              Reporter:
              sushanth Sushanth Sowmyan
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: