Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-11408

HiveServer2 is leaking ClassLoaders when add jar / temporary functions are used due to constructor caching in Hadoop ReflectionUtils

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.13.0, 0.14.0, 0.13.1, 1.0.0, 1.1.1
    • Fix Version/s: 1.1.1
    • Component/s: HiveServer2
    • Labels:
      None

      Description

      I'm able to reproduce with 0.14. I'm yet to see if HIVE-10453 fixes the issue (since it's on top of a larger patch: HIVE-2573 that was added in 1.2).

      Basically, add jar creates a new classloader for loading the classes from the new jar and adds the new classloader to the SessionState object of user's session, making the older one its parent. Creating a temporary function uses the new classloader to load the class used for the function. On closing a session, although there is code to close the classloader for the session, I'm not seeing the new classloader getting GCed and from the heapdump I can see it holds on to the temporary function's class that should have gone away after the session close.

      Steps to reproduce:
      1.

      jdbc:hive2://localhost:10000/> add jar hdfs:///tmp/audf.jar;
      

      2.
      Use a profiler (I'm using yourkit) to verify that a new URLClassLoader was added.

      3.

      jdbc:hive2://localhost:10000/> CREATE TEMPORARY FUNCTION funcA AS 'org.gumashta.udf.AUDF'; 
      

      4.
      Close the jdbc session.

      5.
      Take the memory snapshot and verify that the new URLClassLoader is indeed there and is holding onto the class it loaded (org.gumashta.udf.AUDF) for the session which we already closed.

        Attachments

        1. HIVE-11408.1.patch
          1 kB
          Vaibhav Gumashta
        2. HIVE-11408.2.patch
          6 kB
          Vaibhav Gumashta

          Issue Links

            Activity

              People

              • Assignee:
                vgumashta Vaibhav Gumashta
                Reporter:
                vgumashta Vaibhav Gumashta
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: