Hive
  1. Hive
  2. HIVE-6364

HiveServer2 - Request serving thread should get class loader from existing SessionState

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.13.0
    • Component/s: HiveServer2
    • Labels:
      None

      Description

      SessionState is created for each session in HS2. If we do any add jars, a class loader is set in the SessionState's conf object. This class loader should also be set in each thread that serves request of the same session.

      Scenario (both requests are in the same session)-

      // req 1
      add jar foo.jar // Served by thread th1, this updates class loader and sets in SessionState.conf
      
      // req2 served by th2, such that th1 != th2
      CREATE TEMPORARY FUNCTION foo_udf AS 'some class in foo.jar' 
      // This can throw class not found error, because although 
      // the new thread (th2) gets the same session state as th1,
      // the class loader is different (Thread.currentThread.getContextClassLoader()
      
      

        Issue Links

          Activity

          Hide
          Ashutosh Chauhan added a comment -

          Fixed via HIVE-3969

          Show
          Ashutosh Chauhan added a comment - Fixed via HIVE-3969
          Hide
          Ashutosh Chauhan added a comment -

          I think Navis 's patch on HIVE-3969 is more complete and subsumes current patch. So, I guess we should try to get that one and close this one. Although, Jaideep Dhok 's earlier point about jars on a thread being visible across session is still there. Though, I am not sure how will that negatively impact query execution and way to fix it. That probably can be taken up as follow-up of HIVE-3969

          Show
          Ashutosh Chauhan added a comment - I think Navis 's patch on HIVE-3969 is more complete and subsumes current patch. So, I guess we should try to get that one and close this one. Although, Jaideep Dhok 's earlier point about jars on a thread being visible across session is still there. Though, I am not sure how will that negatively impact query execution and way to fix it. That probably can be taken up as follow-up of HIVE-3969
          Hide
          Jaideep Dhok added a comment -

          Ashutosh Chauhan I will put up a new patch.
          Jason Dere Add jar will always update the class loader. That's the current behaviour. I think the first class loader is set using the conf.getClassLoader method, if nothing is set it will return the default class loader.

          Show
          Jaideep Dhok added a comment - Ashutosh Chauhan I will put up a new patch. Jason Dere Add jar will always update the class loader. That's the current behaviour. I think the first class loader is set using the conf.getClassLoader method, if nothing is set it will return the default class loader.
          Hide
          Jason Dere added a comment -

          Hi Jaideep, when I tried debugging hiveserver2 due to HIVE-6672, it appeared that there was a thread running for each connection (session). Non-SQL commands (such as ADD JAR), were being run within this session thread and so the classloader for the session thread had the JARs loaded. When a SQL command was executed the session thread would start a new thread, and it appeared that this new thread was using the same classloader (and had the added JARs in the classloader's list of URLs). Were you seeing different behavior in your testing (I was running this on Mac, I think with jdk 1.6, not sure if it would have been different)?

          In the patch, the thread's classloader is getting set to the HiveConf's classloader .. where is the HiveConf's classloader getting set from? Do we need to worry about having to make sure this classloader is updated whenever a JAR is added to the classpath?

          Show
          Jason Dere added a comment - Hi Jaideep, when I tried debugging hiveserver2 due to HIVE-6672 , it appeared that there was a thread running for each connection (session). Non-SQL commands (such as ADD JAR), were being run within this session thread and so the classloader for the session thread had the JARs loaded. When a SQL command was executed the session thread would start a new thread, and it appeared that this new thread was using the same classloader (and had the added JARs in the classloader's list of URLs). Were you seeing different behavior in your testing (I was running this on Mac, I think with jdk 1.6, not sure if it would have been different)? In the patch, the thread's classloader is getting set to the HiveConf's classloader .. where is the HiveConf's classloader getting set from? Do we need to worry about having to make sure this classloader is updated whenever a JAR is added to the classpath?
          Hide
          Ashutosh Chauhan added a comment -

          Although, HIVE-3969 is marked as duplicate, I don't think it is a duplicate. This one fixes the problem of having right class loader for a thread serving the query, whereas HIVE-3969 talks about unloading registered jars. So, it seems there are two independent problem, both of which needs to be fixed.
          Jaideep Dhok would you like to rebase your patch.

          Show
          Ashutosh Chauhan added a comment - Although, HIVE-3969 is marked as duplicate, I don't think it is a duplicate. This one fixes the problem of having right class loader for a thread serving the query, whereas HIVE-3969 talks about unloading registered jars. So, it seems there are two independent problem, both of which needs to be fixed. Jaideep Dhok would you like to rebase your patch.
          Hide
          Jaideep Dhok added a comment -

          There's another issue - In SessionState.registerJar class loader is obtained from the current thread. If the current thread was used during another session, then the current session can get some jars from that session.

          Show
          Jaideep Dhok added a comment - There's another issue - In SessionState.registerJar class loader is obtained from the current thread. If the current thread was used during another session, then the current session can get some jars from that session.
          Hide
          Jaideep Dhok added a comment -

          attaching patch

          Show
          Jaideep Dhok added a comment - attaching patch
          Hide
          Jaideep Dhok added a comment -

          Raised review board request - https://reviews.apache.org/r/17708/

          Show
          Jaideep Dhok added a comment - Raised review board request - https://reviews.apache.org/r/17708/

            People

            • Assignee:
              Unassigned
              Reporter:
              Jaideep Dhok
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development