[HADOOP-10513] ReflectionUtils::CONSTRUCTOR_CACHE leaks class loaders

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.4.0
• Component/s: util
• Labels: leak permgen
• Assignee: Unassigned
• Reporter: Gopal Vijayaraghavan
• Created: 16/Apr/14 21:16
• Updated: 30/Jul/15 17:01

Description

Any code which uses custom class-loaders needs to be aware of the ReflectionUtils::CONSTRUCTOR_CACHE

 /** 
   * Cache of constructors for each class. Pins the classes so they
   * can't be garbage collected until ReflectionUtils can be collected.
   */
  private static final Map<Class<?>, Constructor<?>> CONSTRUCTOR_CACHE = 
    new ConcurrentHashMap<Class<?>, Constructor<?>>();

This is not a problem when using only 1 AppClassLoader.

But in cases where the application uses multiple classloaders (to isolate UDFs), this holds onto class references and their associated class loaders by ref (& that leaks all the statics).

The clear method for this cache is unfortunately marked only for testing.

  // methods to support testing
  static void clearCache() {
    CONSTRUCTOR_CACHE.clear();
  }

The cache shows up as the only reference for the class loaders.

Issue Links

• relates to: HADOOP-11771 Configuration::getClassByNameOrNull synchronizes on a static object
• relates to: HIVE-11408 HiveServer2 is leaking ClassLoaders when add jar / temporary functions are used due to constructor caching in Hadoop ReflectionUtils