Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-11499

Datanucleus leaks classloaders when used using embedded metastore with HiveServer2 with UDFs

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.14.0, 1.0.0, 1.2.0, 1.1.0, 1.1.1, 1.2.1
    • Fix Version/s: 1.3.0, 2.0.0
    • Component/s: HiveServer2, Metastore
    • Labels:
      None

      Description

      When UDFs are used, we create a new classloader to add the UDF jar. Similar to what hadoop's reflection utils does(HIVE-11408), datanucleus caches the classloaders (https://github.com/datanucleus/datanucleus-core/blob/3.2/src/java/org/datanucleus/NucleusContext.java#L161). JDOPersistanceManager factory (1 per JVM) holds on to a NucleusContext reference (https://github.com/datanucleus/datanucleus-api-jdo/blob/3.2/src/java/org/datanucleus/api/jdo/JDOPersistenceManagerFactory.java#L115). Until we call NucleusContext#close, the classloader cache is not cleared. In case of UDFs this can lead to permgen leak, as shown in the attached screenshot, where NucleusContext holds on to several URLClassloader objects.

        Attachments

        1. HS2-NucleusCache-Leak.tiff
          833 kB
          Vaibhav Gumashta
        2. HIVE-11499.4.patch
          12 kB
          Vaibhav Gumashta
        3. HIVE-11499.3.patch
          12 kB
          Vaibhav Gumashta
        4. HIVE-11499.1.patch
          3 kB
          Vaibhav Gumashta

          Issue Links

            Activity

              People

              • Assignee:
                vgumashta Vaibhav Gumashta
                Reporter:
                vgumashta Vaibhav Gumashta
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: