[HIVE-11499] Datanucleus leaks classloaders when used using embedded metastore with HiveServer2 with UDFs

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.14.0, 1.0.0, 1.1.0, 1.1.1, 1.2.0, 1.2.1
• Fix Version/s: 1.3.0, 2.0.0
• Component/s: HiveServer2, Metastore
• Assignee: Vaibhav Gumashta
• Reporter: Vaibhav Gumashta
• Created: 07/Aug/15 21:16
• Updated: 15/Mar/17 16:26
• Resolved: 14/Oct/15 01:56

Description

When UDFs are used, we create a new classloader to add the UDF jar. Similar to what hadoop's reflection utils does(HIVE-11408), datanucleus caches the classloaders (https://github.com/datanucleus/datanucleus-core/blob/3.2/src/java/org/datanucleus/NucleusContext.java#L161). JDOPersistanceManager factory (1 per JVM) holds on to a NucleusContext reference (https://github.com/datanucleus/datanucleus-api-jdo/blob/3.2/src/java/org/datanucleus/api/jdo/JDOPersistenceManagerFactory.java#L115). Until we call NucleusContext#close, the classloader cache is not cleared. In case of UDFs this can lead to permgen leak, as shown in the attached screenshot, where NucleusContext holds on to several URLClassloader objects.

Issue Links

• is related to: HIVE-12458 remove identity_udf.jar from source
• is related to: HIVE-16160 OutOfMemoryError: GC overhead limit exceeded on Hiveserver2
• is related to: HIVE-11408 HiveServer2 is leaking ClassLoaders when add jar / temporary functions are used due to constructor caching in Hadoop ReflectionUtils
• links to: Review Board