Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-11499

Datanucleus leaks classloaders when used using embedded metastore with HiveServer2 with UDFs

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.14.0, 1.0.0, 1.1.0, 1.1.1, 1.2.0, 1.2.1
    • 1.3.0, 2.0.0
    • HiveServer2, Metastore
    • None

    Description

      When UDFs are used, we create a new classloader to add the UDF jar. Similar to what hadoop's reflection utils does(HIVE-11408), datanucleus caches the classloaders (https://github.com/datanucleus/datanucleus-core/blob/3.2/src/java/org/datanucleus/NucleusContext.java#L161). JDOPersistanceManager factory (1 per JVM) holds on to a NucleusContext reference (https://github.com/datanucleus/datanucleus-api-jdo/blob/3.2/src/java/org/datanucleus/api/jdo/JDOPersistenceManagerFactory.java#L115). Until we call NucleusContext#close, the classloader cache is not cleared. In case of UDFs this can lead to permgen leak, as shown in the attached screenshot, where NucleusContext holds on to several URLClassloader objects.

      Attachments

        1. HIVE-11499.1.patch
          3 kB
          Vaibhav Gumashta
        2. HIVE-11499.3.patch
          12 kB
          Vaibhav Gumashta
        3. HIVE-11499.4.patch
          12 kB
          Vaibhav Gumashta
        4. HS2-NucleusCache-Leak.tiff
          833 kB
          Vaibhav Gumashta

        Issue Links

          Activity

            People

              vgumashta Vaibhav Gumashta
              vgumashta Vaibhav Gumashta
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: