[HIVE-20192] HS2 with embedded metastore is leaking JDOPersistenceManager objects.

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.0, 3.0.0, 4.0.0
• Fix Version/s: 3.2.0, 4.0.0-alpha-1
• Component/s: HiveServer2
• Labels: HiveServer2 pull-request-available
• Target Version/s: 3.2.0, 4.0.0
• Assignee: Sankar Hariappan
• Reporter: Sankar Hariappan
• Created: 17/Jul/18 09:40
• Updated: 17/Nov/22 08:54
• Resolved: 23/Jul/18 11:38

Description

Hiveserver2 instances where crashing every 3-4 days and observed HS2 in on unresponsive state. Also, observed that the FGC collection happening regularly

From JXray report it is seen that pmCache(List of JDOPersistenceManager objects) is occupying 84% of the heap and there are around 16,000 references of UDFClassLoader.

10,759,230K (84.7%) Object tree for GC root(s) Java Static org.apache.hadoop.hive.metastore.ObjectStore.pmf
- org.datanucleus.api.jdo.JDOPersistenceManagerFactory.pmCache ↘ 10,744,419K (84.6%), 1 reference(s)
  - j.u.Collections$SetFromMap.m ↘ 10,744,419K (84.6%), 1 reference(s)
    - {java.util.concurrent.ConcurrentHashMap}.keys ↘ 10,743,764K (84.5%), 16,872 reference(s)
      - org.datanucleus.api.jdo.JDOPersistenceManager.ec ↘ 10,738,831K (84.5%), 16,872 reference(s)
        ... 3 more references together retaining 4,933K (< 0.1%)
    - java.util.concurrent.ConcurrentHashMap self 655K (< 0.1%), 1 object(s)
      ... 2 more references together retaining 48b (< 0.1%)
- org.datanucleus.api.jdo.JDOPersistenceManagerFactory.nucleusContext ↘ 14,810K (0.1%), 1 reference(s)
... 3 more references together retaining 96b (< 0.1%)

When the RawStore object is re-created, it is not allowed to be updated into the ThreadWithGarbageCleanup.threadRawStoreMap which leads to the new RawStore never gets cleaned-up when the thread exit.

 

Issue Links

• links to: GitHub Pull Request #402