Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-20192

HS2 with embedded metastore is leaking JDOPersistenceManager objects.

Log workAgile BoardRank to TopRank to BottomVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

      Description

      Hiveserver2 instances where crashing every 3-4 days and observed HS2 in on unresponsive state. Also, observed that the FGC collection happening regularly

      From JXray report it is seen that pmCache(List of JDOPersistenceManager objects) is occupying 84% of the heap and there are around 16,000 references of UDFClassLoader.

      10,759,230K (84.7%) Object tree for GC root(s) Java Static org.apache.hadoop.hive.metastore.ObjectStore.pmf
      - org.datanucleus.api.jdo.JDOPersistenceManagerFactory.pmCache ↘ 10,744,419K (84.6%), 1 reference(s)
        - j.u.Collections$SetFromMap.m ↘ 10,744,419K (84.6%), 1 reference(s)
          - {java.util.concurrent.ConcurrentHashMap}.keys ↘ 10,743,764K (84.5%), 16,872 reference(s)
            - org.datanucleus.api.jdo.JDOPersistenceManager.ec ↘ 10,738,831K (84.5%), 16,872 reference(s)
              ... 3 more references together retaining 4,933K (< 0.1%)
          - java.util.concurrent.ConcurrentHashMap self 655K (< 0.1%), 1 object(s)
            ... 2 more references together retaining 48b (< 0.1%)
      - org.datanucleus.api.jdo.JDOPersistenceManagerFactory.nucleusContext ↘ 14,810K (0.1%), 1 reference(s)
      ... 3 more references together retaining 96b (< 0.1%)

      When the RawStore object is re-created, it is not allowed to be updated into the ThreadWithGarbageCleanup.threadRawStoreMap which leads to the new RawStore never gets cleaned-up when the thread exit.

       

        Attachments

        1. HIVE-20192.01.patch
          4 kB
          Sankar Hariappan
        2. HIVE-20192.01-branch-3.patch
          4 kB
          Sankar Hariappan
        3. HIVE-20192.02.patch
          4 kB
          Sankar Hariappan

          Activity

          $i18n.getText('security.level.explanation', $currentSelection) Viewable by All Users
          Cancel

            People

            • Assignee:
              sankarh Sankar Hariappan Assign to me
              Reporter:
              sankarh Sankar Hariappan

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment