[HIVE-24858] UDFClassLoader leak in Configuration.CACHE_CLASSES

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.0.0-alpha-1
• Labels: pull-request-available
• Assignee: Karen Coppage
• Reporter: Karen Coppage
• Created: 08/Mar/21 14:47
• Updated: 03/Sep/24 08:31
• Resolved: 06/Jul/21 09:06

Description

If a UDF jar has been registered in a session and a temporary function created from it, when the session is closed its UDFClassLoader is not GC'd as it has been leaked to the session's HiveConf object's cache. Since the ClassLoader is not GC'd, the UDF jar's classes aren't GC'd from Metaspace. This can potentially lead to Metaspace OOM.
Path to GC root is:

Class Name                                                                                                                          | Shallow Heap | Retained Heap
-------------------------------------------------------------------------------------------------------------------------------------------------------------------
contextClassLoader org.apache.hive.service.server.ThreadWithGarbageCleanup @ 0x7164deb50  HiveServer2-Handler-Pool: Thread-72 Thread|          128 |        79,072
referent java.util.WeakHashMap$Entry @ 0x7164e67d0                                                                                  |           40 |           824
'- [6] java.util.WeakHashMap$Entry[16] @ 0x71581aac0                                                                                |           80 |         5,056
   '- table java.util.WeakHashMap @ 0x71580f510                                                                                     |           48 |         6,920
      '- CACHE_CLASSES class org.apache.hadoop.conf.Configuration @ 0x71580f3d8                                                     |           64 |        74,528
-------------------------------------------------------------------------------------------------------------------------------------------------------------------

Issue Links

• links to: GitHub Pull Request #2049