Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-11402

HS2 - add an option to disallow parallel query execution within a single Session

    Details

      Description

      HiveServer2 currently allows concurrent queries to be run in a single session. However, every HS2 session has an associated SessionState object, and the use of SessionState in many places assumes that only one thread is using it, ie it is not thread safe.
      There are many places where SesssionState thread safety needs to be addressed, and until then we should serialize all query execution for a single HS2 session. This problem can become more visible with HIVE-4239 now allowing parallel query compilation.

      Note that running queries in parallel for single session is not straightforward with jdbc, you need to spawn another thread as the Statement.execute calls are blocking. I believe ODBC has non blocking query execution API, and Hue is another well known application that shares sessions for all queries that a user runs.

      1. HIVE-11402.patch
        14 kB
        Sergey Shelukhin
      2. HIVE-11402.01.patch
        15 kB
        Sergey Shelukhin
      3. HIVE-11402.02.patch
        15 kB
        Sergey Shelukhin
      4. HIVE-11402.03.patch
        30 kB
        Sergey Shelukhin
      5. HIVE-11402.04.patch
        31 kB
        Sergey Shelukhin
      6. HIVE-11402.05.patch
        31 kB
        Sergey Shelukhin
      7. HIVE-11402.06.patch
        31 kB
        Sergey Shelukhin

        Issue Links

          Activity

          Hide
          thejas Thejas M Nair added a comment -

          Comment from earlier discussion with Vaibhav Gumashta and Jason Dere - In Hive SessionState since there are several fields that are associated with the currently executing query, such as commandType, lastCommand. Also SessionState has a Conf object and this also gets set with query-specific information.

          Show
          thejas Thejas M Nair added a comment - Comment from earlier discussion with Vaibhav Gumashta and Jason Dere - In Hive SessionState since there are several fields that are associated with the currently executing query, such as commandType, lastCommand. Also SessionState has a Conf object and this also gets set with query-specific information.
          Hide
          thejas Thejas M Nair added a comment -

          From Eugene Koifman - Each SessionState has 1 instance of HiveTxnManager which can only handle 1 txn at a time.
          While JDBC spec doesn't make concurrency guarantees explicit, most info on the web says it's a bad idea to allow this.
          For example, suppose you setAutoCommit(false) and then run to statements concurrently on this Connection, would you expect 1 transaction for each statement or 1 txn for both.

          Show
          thejas Thejas M Nair added a comment - From Eugene Koifman - Each SessionState has 1 instance of HiveTxnManager which can only handle 1 txn at a time. While JDBC spec doesn't make concurrency guarantees explicit, most info on the web says it's a bad idea to allow this. For example, suppose you setAutoCommit(false) and then run to statements concurrently on this Connection, would you expect 1 transaction for each statement or 1 txn for both.
          Hide
          sershe Sergey Shelukhin added a comment -

          HIVE-4239 only allows parallel compilation between sessions

          Show
          sershe Sergey Shelukhin added a comment - HIVE-4239 only allows parallel compilation between sessions
          Hide
          thejas Thejas M Nair added a comment -

          Sergey Shelukhin Thanks for clarifying, I had forgotten about the SessionState.get().getCompileLock() that you added. I will update the description.
          Looks like we can still have issues outside of compilation, during parallel query execution within session.

          Show
          thejas Thejas M Nair added a comment - Sergey Shelukhin Thanks for clarifying, I had forgotten about the SessionState.get().getCompileLock() that you added. I will update the description. Looks like we can still have issues outside of compilation, during parallel query execution within session.
          Hide
          cwsteinbach Carl Steinbach added a comment -

          Note that running queries in parallel for single session is not straightforward with jdbc, you need to spawn another thread as the Statement.execute calls are blocking. I believe ODBC has non blocking query execution API ...

          I made a couple mistakes when I designed the HS2 API (the use of Thrift Enums and Unions comes to mind), but by far the biggest mistake was allowing a 1:many relationship between Sessions and Operations. At the time I thought there was a chance that the ODBC spec required this, but now think this is something best handled on the client side. Providing support for the 1:many Session:Operation mapping results in a lot of additional complexity on the server-side, only to yield a feature with a very high potential for misuse.

          Rather than temporarily serializing operations against a given session, I propose instead that we enforce a 1:1 mapping between HS2 sessions and active operations. This is a backward incompatible change, but one which I think will yield far better results in the long term.

          Show
          cwsteinbach Carl Steinbach added a comment - Note that running queries in parallel for single session is not straightforward with jdbc, you need to spawn another thread as the Statement.execute calls are blocking. I believe ODBC has non blocking query execution API ... I made a couple mistakes when I designed the HS2 API (the use of Thrift Enums and Unions comes to mind), but by far the biggest mistake was allowing a 1:many relationship between Sessions and Operations. At the time I thought there was a chance that the ODBC spec required this, but now think this is something best handled on the client side. Providing support for the 1:many Session:Operation mapping results in a lot of additional complexity on the server-side, only to yield a feature with a very high potential for misuse. Rather than temporarily serializing operations against a given session, I propose instead that we enforce a 1:1 mapping between HS2 sessions and active operations. This is a backward incompatible change, but one which I think will yield far better results in the long term.
          Hide
          sershe Sergey Shelukhin added a comment -

          Simple patch that adds a semaphore to acquire/release methods, configurable and on by default (parallel operations are off, rather). Thejas M Nair can you take a look? Esp. wrt what is an operation and what isn't, e.g. should fetching results be allowed to run in parallel with other things?

          I will post an RB shortly

          Show
          sershe Sergey Shelukhin added a comment - Simple patch that adds a semaphore to acquire/release methods, configurable and on by default (parallel operations are off, rather). Thejas M Nair can you take a look? Esp. wrt what is an operation and what isn't, e.g. should fetching results be allowed to run in parallel with other things? I will post an RB shortly
          Hide
          cwsteinbach Carl Steinbach added a comment -

          HiveSessionImpl is starting to look a lot like SessionState. I suppose that was inevitable

          Show
          cwsteinbach Carl Steinbach added a comment - HiveSessionImpl is starting to look a lot like SessionState. I suppose that was inevitable
          Hide
          hiveqa Hive QA added a comment -

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12813993/HIVE-11402.01.patch

          SUCCESS: +1 due to 1 test(s) being added or modified.

          ERROR: -1 due to 7 failed/errored test(s), 10285 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_part
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
          org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
          org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
          org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testDelayedLocalityNodeCommErrorImmediateAllocation
          

          Test results: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/285/testReport
          Console output: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/285/console
          Test logs: http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-285/

          Messages:

          Executing org.apache.hive.ptest.execution.TestCheckPhase
          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 7 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12813993 - PreCommit-HIVE-MASTER-Build

          Show
          hiveqa Hive QA added a comment - Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12813993/HIVE-11402.01.patch SUCCESS: +1 due to 1 test(s) being added or modified. ERROR: -1 due to 7 failed/errored test(s), 10285 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_part org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testDelayedLocalityNodeCommErrorImmediateAllocation Test results: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/285/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/285/console Test logs: http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-285/ Messages: Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 7 tests failed This message is automatically generated. ATTACHMENT ID: 12813993 - PreCommit-HIVE-MASTER-Build
          Hide
          sershe Sergey Shelukhin added a comment -

          Test failures are unrelated

          Show
          sershe Sergey Shelukhin added a comment - Test failures are unrelated
          Hide
          mohitsabharwal Mohit Sabharwal added a comment -

          Sergey Shelukhin, we should keep the default for hive.server2.parallel.ops.in.session as true, otherwise this is a backward incompatible change for Hue and any other client code where queries currently share sessions.

          Show
          mohitsabharwal Mohit Sabharwal added a comment - Sergey Shelukhin , we should keep the default for hive.server2.parallel.ops.in.session as true, otherwise this is a backward incompatible change for Hue and any other client code where queries currently share sessions.
          Hide
          sershe Sergey Shelukhin added a comment -

          Fixed.

          Show
          sershe Sergey Shelukhin added a comment - Fixed.
          Hide
          hiveqa Hive QA added a comment -

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12815513/HIVE-11402.02.patch

          SUCCESS: +1 due to 1 test(s) being added or modified.

          ERROR: -1 due to 7 failed/errored test(s), 10287 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
          org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
          org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
          org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testDelayedLocalityNodeCommErrorImmediateAllocation
          org.apache.hive.spark.client.TestSparkClient.testJobSubmission
          

          Test results: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/335/testReport
          Console output: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/335/console
          Test logs: http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-335/

          Messages:

          Executing org.apache.hive.ptest.execution.TestCheckPhase
          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 7 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12815513 - PreCommit-HIVE-MASTER-Build

          Show
          hiveqa Hive QA added a comment - Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12815513/HIVE-11402.02.patch SUCCESS: +1 due to 1 test(s) being added or modified. ERROR: -1 due to 7 failed/errored test(s), 10287 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testDelayedLocalityNodeCommErrorImmediateAllocation org.apache.hive.spark.client.TestSparkClient.testJobSubmission Test results: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/335/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/335/console Test logs: http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-335/ Messages: Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 7 tests failed This message is automatically generated. ATTACHMENT ID: 12815513 - PreCommit-HIVE-MASTER-Build
          Hide
          aihuaxu Aihua Xu added a comment -

          Sergey Shelukhin Do you still observe parallel operation issues in the single session? I have separated QueryState out of SessionState in HIVE-13424. From previous comments, seems it's also common to share operations in the same session like in HUE. Should we continue to fix parallel issue if we have any rather than disallowing parallel execution?

          Show
          aihuaxu Aihua Xu added a comment - Sergey Shelukhin Do you still observe parallel operation issues in the single session? I have separated QueryState out of SessionState in HIVE-13424 . From previous comments, seems it's also common to share operations in the same session like in HUE. Should we continue to fix parallel issue if we have any rather than disallowing parallel execution?
          Hide
          sershe Sergey Shelukhin added a comment -

          This patch has been changed to allow it by default. This is just a safety flag if someone sees issues.
          Last time I checked, the runtime parallel operations in the same session work purely by magic (I think I left a comment somewhere).
          I.e. I believe they may work but there's no good reason why they do, because non-thread-safe objects appear to be used without synchronization; there may be some bugs.

          cc Thejas M Nair

          Show
          sershe Sergey Shelukhin added a comment - This patch has been changed to allow it by default. This is just a safety flag if someone sees issues. Last time I checked, the runtime parallel operations in the same session work purely by magic (I think I left a comment somewhere). I.e. I believe they may work but there's no good reason why they do, because non-thread-safe objects appear to be used without synchronization; there may be some bugs. cc Thejas M Nair
          Hide
          aihuaxu Aihua Xu added a comment -

          hmm. Yeah. Saw your comments in HiveSessionImpl and we do have issues that some variables in SessionState are not protected for multiple threads.

          Seems to make sense to have such option and continue fixing the thread-safe issue. Just wondering if the option is set to false, what kind of effect to the new operation in the same session? will it just get blocked to get the previous one finished?

          Show
          aihuaxu Aihua Xu added a comment - hmm. Yeah. Saw your comments in HiveSessionImpl and we do have issues that some variables in SessionState are not protected for multiple threads. Seems to make sense to have such option and continue fixing the thread-safe issue. Just wondering if the option is set to false, what kind of effect to the new operation in the same session? will it just get blocked to get the previous one finished?
          Hide
          sershe Sergey Shelukhin added a comment -

          Yeah, there's a single-entry semaphore that controls the execution of most operations (not e.g. fetchresults, or cancel operation )

          Show
          sershe Sergey Shelukhin added a comment - Yeah, there's a single-entry semaphore that controls the execution of most operations (not e.g. fetchresults, or cancel operation )
          Hide
          sershe Sergey Shelukhin added a comment -

          Aihua Xu do you have more input on the patch?

          Show
          sershe Sergey Shelukhin added a comment - Aihua Xu do you have more input on the patch?
          Hide
          aihuaxu Aihua Xu added a comment -

          Just worried how useful that configuration will be since mostly the users will use mixed HUE, beeline, etc, I guess. So we may have to keep the default. But seems this is safe to add.

          +1.

          Show
          aihuaxu Aihua Xu added a comment - Just worried how useful that configuration will be since mostly the users will use mixed HUE, beeline, etc, I guess. So we may have to keep the default. But seems this is safe to add. +1.
          Hide
          thejas Thejas M Nair added a comment -

          Sorry about the delay in reviewing.
          This will however not disallow query execution in parallel when the query execution is done asynchronously (the default mode with our jdbc driver).

          Show
          thejas Thejas M Nair added a comment - Sorry about the delay in reviewing. This will however not disallow query execution in parallel when the query execution is done asynchronously (the default mode with our jdbc driver).
          Hide
          thejas Thejas M Nair added a comment - - edited

          Maybe pass the lock object to operation async execution and have it release it when its done ?
          ie, send it all way to SQLOperation.runInternal ..

          Show
          thejas Thejas M Nair added a comment - - edited Maybe pass the lock object to operation async execution and have it release it when its done ? ie, send it all way to SQLOperation.runInternal ..
          Hide
          sershe Sergey Shelukhin added a comment -

          The handling for async ops, as well as some refactoring. Thanks for the pointer!

          Show
          sershe Sergey Shelukhin added a comment - The handling for async ops, as well as some refactoring. Thanks for the pointer!
          Hide
          hiveqa Hive QA added a comment -

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12817230/HIVE-11402.03.patch

          SUCCESS: +1 due to 1 test(s) being added or modified.

          ERROR: -1 due to 9 failed/errored test(s), 10294 tests executed
          Failed tests:

          TestMiniTezCliDriver-tez_joins_explain.q-vector_data_types.q-tez_dynpart_hashjoin_1.q-and-12-more - did not produce a TEST-*.xml file
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
          org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
          org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.org.apache.hadoop.hive.cli.TestMinimrCliDriver
          

          Test results: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/482/testReport
          Console output: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/482/console
          Test logs: http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-482/

          Messages:

          Executing org.apache.hive.ptest.execution.TestCheckPhase
          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 9 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12817230 - PreCommit-HIVE-MASTER-Build

          Show
          hiveqa Hive QA added a comment - Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12817230/HIVE-11402.03.patch SUCCESS: +1 due to 1 test(s) being added or modified. ERROR: -1 due to 9 failed/errored test(s), 10294 tests executed Failed tests: TestMiniTezCliDriver-tez_joins_explain.q-vector_data_types.q-tez_dynpart_hashjoin_1.q-and-12-more - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit org.apache.hadoop.hive.cli.TestMinimrCliDriver.org.apache.hadoop.hive.cli.TestMinimrCliDriver Test results: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/482/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/482/console Test logs: http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-482/ Messages: Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 9 tests failed This message is automatically generated. ATTACHMENT ID: 12817230 - PreCommit-HIVE-MASTER-Build
          Hide
          thejas Thejas M Nair added a comment -

          +1 Changes look good. Please verify the UT failures are not related.

          Show
          thejas Thejas M Nair added a comment - +1 Changes look good. Please verify the UT failures are not related.
          Hide
          thejas Thejas M Nair added a comment -

          Sergey Shelukhin Can you also add a simple test to ensure that the queries get serialized ? (So we know this feature works )

          Something like -
          HiveStatement stmt1 = (HiveStatement) con.createStatement();
          HiveStatement stmt2 = (HiveStatement) con.createStatement();
          stmt1.executeAsync(create table t(i int));
          stmt1.executeAsync(insert into t select sleepUDF(1))
          stmt2.executeAsync(select * from t )
          verify above statement returned 1 row

          Show
          thejas Thejas M Nair added a comment - Sergey Shelukhin Can you also add a simple test to ensure that the queries get serialized ? (So we know this feature works ) Something like - HiveStatement stmt1 = (HiveStatement) con.createStatement(); HiveStatement stmt2 = (HiveStatement) con.createStatement(); stmt1.executeAsync(create table t(i int)); stmt1.executeAsync(insert into t select sleepUDF(1)) stmt2.executeAsync(select * from t ) verify above statement returned 1 row
          Hide
          thejas Thejas M Nair added a comment -

          See TestJdbcDriver2 for example. We would ofcourse need the config to be set to prevent parallel query in the test suite.

          Show
          thejas Thejas M Nair added a comment - See TestJdbcDriver2 for example. We would ofcourse need the config to be set to prevent parallel query in the test suite.
          Hide
          sershe Sergey Shelukhin added a comment -

          Added the test as suggested. I am not familiar with the setup of this test so I wouldn't vouch for its relevance

          Show
          sershe Sergey Shelukhin added a comment - Added the test as suggested. I am not familiar with the setup of this test so I wouldn't vouch for its relevance
          Hide
          sershe Sergey Shelukhin added a comment -

          Also rebased the patch.

          Show
          sershe Sergey Shelukhin added a comment - Also rebased the patch.
          Hide
          thejas Thejas M Nair added a comment -

          Test case looks good. (though i would have preferred the use of a sleepUDF to guarantee how long the first query would take).
          Just some minor comments about comments in reviewboard.
          +1 with the review comments addressed.

          Show
          thejas Thejas M Nair added a comment - Test case looks good. (though i would have preferred the use of a sleepUDF to guarantee how long the first query would take). Just some minor comments about comments in reviewboard. +1 with the review comments addressed.
          Hide
          sershe Sergey Shelukhin added a comment -

          Changed the test to use sleep UDF, and sleep for half a second

          Show
          sershe Sergey Shelukhin added a comment - Changed the test to use sleep UDF, and sleep for half a second
          Hide
          sershe Sergey Shelukhin added a comment -

          Will commit after HiveQA

          Show
          sershe Sergey Shelukhin added a comment - Will commit after HiveQA
          Hide
          hiveqa Hive QA added a comment -

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12817855/HIVE-11402.06.patch

          SUCCESS: +1 due to 2 test(s) being added or modified.

          ERROR: -1 due to 9 failed/errored test(s), 10319 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.org.apache.hadoop.hive.cli.TestMinimrCliDriver
          org.apache.hadoop.hive.llap.daemon.impl.TestLlapTokenChecker.testCheckPermissions
          org.apache.hadoop.hive.llap.daemon.impl.TestLlapTokenChecker.testGetToken
          org.apache.hadoop.hive.llap.security.TestLlapSignerImpl.testSigning
          

          Test results: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/508/testReport
          Console output: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/508/console
          Test logs: http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-508/

          Messages:

          Executing org.apache.hive.ptest.execution.TestCheckPhase
          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 9 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12817855 - PreCommit-HIVE-MASTER-Build

          Show
          hiveqa Hive QA added a comment - Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12817855/HIVE-11402.06.patch SUCCESS: +1 due to 2 test(s) being added or modified. ERROR: -1 due to 9 failed/errored test(s), 10319 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit org.apache.hadoop.hive.cli.TestMinimrCliDriver.org.apache.hadoop.hive.cli.TestMinimrCliDriver org.apache.hadoop.hive.llap.daemon.impl.TestLlapTokenChecker.testCheckPermissions org.apache.hadoop.hive.llap.daemon.impl.TestLlapTokenChecker.testGetToken org.apache.hadoop.hive.llap.security.TestLlapSignerImpl.testSigning Test results: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/508/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/508/console Test logs: http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-508/ Messages: Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 9 tests failed This message is automatically generated. ATTACHMENT ID: 12817855 - PreCommit-HIVE-MASTER-Build
          Hide
          thejas Thejas M Nair added a comment -

          Committed the patch. Verified that the failures are not related.

          Show
          thejas Thejas M Nair added a comment - Committed the patch. Verified that the failures are not related.
          Hide
          leftylev Lefty Leverenz added a comment -

          Doc note: This adds hive.server2.parallel.ops.in.session to HiveConf.java, so it will need to be documented in the wiki for releases 2.1.1 and 2.2.0.

          Added TODOC2.1.1 and TODOC2.2 labels.

          Show
          leftylev Lefty Leverenz added a comment - Doc note: This adds hive.server2.parallel.ops.in.session to HiveConf.java, so it will need to be documented in the wiki for releases 2.1.1 and 2.2.0. Configuration Properties – HiveServer2 Added TODOC2.1.1 and TODOC2.2 labels.

            People

            • Assignee:
              sershe Sergey Shelukhin
              Reporter:
              thejas Thejas M Nair
            • Votes:
              0 Vote for this issue
              Watchers:
              22 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development