Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-20192

HS2 with embedded metastore is leaking JDOPersistenceManager objects.

    Details

      Description

      Hiveserver2 instances where crashing every 3-4 days and observed HS2 in on unresponsive state. Also, observed that the FGC collection happening regularly

      From JXray report it is seen that pmCache(List of JDOPersistenceManager objects) is occupying 84% of the heap and there are around 16,000 references of UDFClassLoader.

      10,759,230K (84.7%) Object tree for GC root(s) Java Static org.apache.hadoop.hive.metastore.ObjectStore.pmf
      - org.datanucleus.api.jdo.JDOPersistenceManagerFactory.pmCache ↘ 10,744,419K (84.6%), 1 reference(s)
        - j.u.Collections$SetFromMap.m ↘ 10,744,419K (84.6%), 1 reference(s)
          - {java.util.concurrent.ConcurrentHashMap}.keys ↘ 10,743,764K (84.5%), 16,872 reference(s)
            - org.datanucleus.api.jdo.JDOPersistenceManager.ec ↘ 10,738,831K (84.5%), 16,872 reference(s)
              ... 3 more references together retaining 4,933K (< 0.1%)
          - java.util.concurrent.ConcurrentHashMap self 655K (< 0.1%), 1 object(s)
            ... 2 more references together retaining 48b (< 0.1%)
      - org.datanucleus.api.jdo.JDOPersistenceManagerFactory.nucleusContext ↘ 14,810K (0.1%), 1 reference(s)
      ... 3 more references together retaining 96b (< 0.1%)

      When the RawStore object is re-created, it is not allowed to be updated into the ThreadWithGarbageCleanup.threadRawStoreMap which leads to the new RawStore never gets cleaned-up when the thread exit.

       

        Attachments

        1. HIVE-20192.01-branch-3.patch
          4 kB
          Sankar Hariappan
        2. HIVE-20192.02.patch
          4 kB
          Sankar Hariappan
        3. HIVE-20192.01.patch
          4 kB
          Sankar Hariappan

          Issue Links

            Activity

              People

              • Assignee:
                sankarh Sankar Hariappan
                Reporter:
                sankarh Sankar Hariappan
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: