Hive
  1. Hive
  2. HIVE-4617 Asynchronous execution in HiveServer2 to run a query in non-blocking mode
  3. HIVE-5217

Add long polling to asynchronous execution in HiveServer2

        Details

        • Type: Sub-task Sub-task
        • Status: Resolved
        • Priority: Major Major
        • Resolution: Fixed
        • Affects Version/s: 0.13.0
        • Fix Version/s: 0.13.0
        • Component/s: HiveServer2
        • Labels:
          None
        • Release Note:
          Hide
          Change to use Long polling as described in description.
          Adds hive.server2.long.polling.timeout configuration parameter, which can be used to configure how long the long poll waits. Most users would not need to bother about changing this configuration parameter.
          Show
          Change to use Long polling as described in description. Adds hive.server2.long.polling.timeout configuration parameter, which can be used to configure how long the long poll waits. Most users would not need to bother about changing this configuration parameter.

          Description

          HIVE-4617 provides support for async execution in HS2. The client gets an operation handle which it can poll to check on the operation status. However, the polling frequency is entirely left to the client which can be resource inefficient. Long polling will solve this, by blocking the client request to check the operation status for a configurable amount of time (a new HS2 config) if the data is not available, but responding immediately if the data is available.

          1. Text File
            HIVE-5217.D12801.6.patch
            8 kB
            Vaibhav Gumashta
          2. Text File
            HIVE-5217.D12801.5.patch
            8 kB
            Vaibhav Gumashta
          3. Text File
            HIVE-5217.D12801.4.patch
            8 kB
            Vaibhav Gumashta
          4. Text File
            HIVE-5217.D12801.3.patch
            5 kB
            Vaibhav Gumashta
          5. Text File
            HIVE-5217.D12801.2.patch
            5 kB
            Vaibhav Gumashta
          6. Text File
            HIVE-5217.6.patch
            28 kB
            Vaibhav Gumashta
          7. Text File
            HIVE-5217.5.patch
            25 kB
            Vaibhav Gumashta
          8. Text File
            HIVE-5217.4.patch
            27 kB
            Vaibhav Gumashta
          9. Text File
            HIVE-5217.3.patch
            17 kB
            Vaibhav Gumashta
          10. Text File
            HIVE-5217.2.patch
            19 kB
            Vaibhav Gumashta

            Issue Links

            blocks

            Sub-task - The sub-task of the issue HIVE-5232 Make JDBC use the new HiveServer2 async execution API by default

            • Major - Major loss of function.
            • Resolved
            is blocked by

            Improvement - An improvement or enhancement to an existing feature or task. HIVE-4617 Asynchronous execution in HiveServer2 to run a query in non-blocking mode

            • Major - Major loss of function.
            • Closed
            relates to

            Sub-task - The sub-task of the issue HIVE-5230 Better error reporting by async threads in HiveServer2

            • Major - Major loss of function.
            • Resolved

              Activity

              Descending order - Click to sort in ascending order
              Hide
              Permalink
              Lefty Leverenz added a comment -

              hive.server2.long.polling.timeout is documented in the wiki here:

              Show
              Lefty Leverenz added a comment - hive.server2.long.polling.timeout is documented in the wiki here: Configuration Properties: hive.server2.long.polling.timeout
              Hide
              Permalink
              Thejas M Nair added a comment -

              Patch committed to trunk.
              Thanks for the contribution Vaibhav!

              Can you add see if the release note looks OK ? Please feel free to edit it.

              Show
              Thejas M Nair added a comment - Patch committed to trunk. Thanks for the contribution Vaibhav! Can you add see if the release note looks OK ? Please feel free to edit it.
              Hide
              Permalink
              Hive QA added a comment -

              Overall: -1 at least one tests failed

              Here are the results of testing the latest attachment:
              https://issues.apache.org/jira/secure/attachment/12630053/HIVE-5217.6.patch

              ERROR: -1 due to 2 failed/errored test(s), 5173 tests executed
              Failed tests:

              org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
org.apache.hcatalog.hbase.TestHBaseDirectOutputFormat.org.apache.hcatalog.hbase.TestHBaseDirectOutputFormat

              Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1439/testReport
              Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1439/console

              Messages:

              Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed

              This message is automatically generated.

              ATTACHMENT ID: 12630053

              Show
              Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12630053/HIVE-5217.6.patch ERROR: -1 due to 2 failed/errored test(s), 5173 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table org.apache.hcatalog.hbase.TestHBaseDirectOutputFormat.org.apache.hcatalog.hbase.TestHBaseDirectOutputFormat Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1439/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1439/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed This message is automatically generated. ATTACHMENT ID: 12630053
              Hide
              Permalink
              Thejas M Nair added a comment -

              +1

              Show
              Thejas M Nair added a comment - +1
              Hide
              Permalink
              Vaibhav Gumashta added a comment -

              Incorporating Thejas' feedback given on RB.

              Show
              Vaibhav Gumashta added a comment - Incorporating Thejas' feedback given on RB.
              Hide
              Permalink
              Hive QA added a comment -

              Overall: -1 no tests executed

              Here are the results of testing the latest attachment:
              https://issues.apache.org/jira/secure/attachment/12629519/HIVE-5217.5.patch

              Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1383/testReport
              Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1383/console

              Messages:

              **** This message was trimmed, see log for full details ****
[INFO] --- maven-install-plugin:2.4:install (default-install) @ hive-shims-0.20S ---
[INFO] Installing /data/hive-ptest/working/apache-svn-trunk-source/shims/0.20S/target/hive-shims-0.20S-0.13.0-SNAPSHOT.jar to /data/hive-ptest/working/maven/org/apache/hive/shims/hive-shims-0.20S/0.13.0-SNAPSHOT/hive-shims-0.20S-0.13.0-SNAPSHOT.jar
[INFO] Installing /data/hive-ptest/working/apache-svn-trunk-source/shims/0.20S/pom.xml to /data/hive-ptest/working/maven/org/apache/hive/shims/hive-shims-0.20S/0.13.0-SNAPSHOT/hive-shims-0.20S-0.13.0-SNAPSHOT.pom
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building Hive Shims 0.23 0.13.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-shims-0.23 ---
[INFO] Deleting /data/hive-ptest/working/apache-svn-trunk-source/shims/0.23 (includes = [datanucleus.log, derby.log], excludes = [])
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ hive-shims-0.23 ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /data/hive-ptest/working/apache-svn-trunk-source/shims/0.23/src/main/resources
[INFO] 
[INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @ hive-shims-0.23 ---
[INFO] Executing tasks

main:
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hive-shims-0.23 ---
[INFO] Compiling 3 source files to /data/hive-ptest/working/apache-svn-trunk-source/shims/0.23/target/classes
[WARNING] Note: /data/hive-ptest/working/apache-svn-trunk-source/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java uses or overrides a deprecated API.
[WARNING] Note: Recompile with -Xlint:deprecation for details.
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ hive-shims-0.23 ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /data/hive-ptest/working/apache-svn-trunk-source/shims/0.23/src/test/resources
[INFO] 
[INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ hive-shims-0.23 ---
[INFO] Executing tasks

main:
    [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/shims/0.23/target/tmp
    [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/shims/0.23/target/warehouse
    [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/shims/0.23/target/tmp/conf
     [copy] Copying 5 files to /data/hive-ptest/working/apache-svn-trunk-source/shims/0.23/target/tmp/conf
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ hive-shims-0.23 ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-surefire-plugin:2.16:test (default-test) @ hive-shims-0.23 ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ hive-shims-0.23 ---
[INFO] Building jar: /data/hive-ptest/working/apache-svn-trunk-source/shims/0.23/target/hive-shims-0.23-0.13.0-SNAPSHOT.jar
[INFO] 
[INFO] --- maven-install-plugin:2.4:install (default-install) @ hive-shims-0.23 ---
[INFO] Installing /data/hive-ptest/working/apache-svn-trunk-source/shims/0.23/target/hive-shims-0.23-0.13.0-SNAPSHOT.jar to /data/hive-ptest/working/maven/org/apache/hive/shims/hive-shims-0.23/0.13.0-SNAPSHOT/hive-shims-0.23-0.13.0-SNAPSHOT.jar
[INFO] Installing /data/hive-ptest/working/apache-svn-trunk-source/shims/0.23/pom.xml to /data/hive-ptest/working/maven/org/apache/hive/shims/hive-shims-0.23/0.13.0-SNAPSHOT/hive-shims-0.23-0.13.0-SNAPSHOT.pom
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building Hive Shims 0.13.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-shims ---
[INFO] Deleting /data/hive-ptest/working/apache-svn-trunk-source/shims/aggregator (includes = [datanucleus.log, derby.log], excludes = [])
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ hive-shims ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /data/hive-ptest/working/apache-svn-trunk-source/shims/aggregator/src/main/resources
[INFO] 
[INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @ hive-shims ---
[INFO] Executing tasks

main:
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hive-shims ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ hive-shims ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /data/hive-ptest/working/apache-svn-trunk-source/shims/aggregator/src/test/resources
[INFO] 
[INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ hive-shims ---
[INFO] Executing tasks

main:
    [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/shims/aggregator/target/tmp
    [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/shims/aggregator/target/warehouse
    [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/shims/aggregator/target/tmp/conf
     [copy] Copying 5 files to /data/hive-ptest/working/apache-svn-trunk-source/shims/aggregator/target/tmp/conf
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ hive-shims ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-surefire-plugin:2.16:test (default-test) @ hive-shims ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ hive-shims ---
[WARNING] JAR will be empty - no content was marked for inclusion!
[INFO] Building jar: /data/hive-ptest/working/apache-svn-trunk-source/shims/aggregator/target/hive-shims-0.13.0-SNAPSHOT.jar
[INFO] 
[INFO] --- maven-install-plugin:2.4:install (default-install) @ hive-shims ---
[INFO] Installing /data/hive-ptest/working/apache-svn-trunk-source/shims/aggregator/target/hive-shims-0.13.0-SNAPSHOT.jar to /data/hive-ptest/working/maven/org/apache/hive/hive-shims/0.13.0-SNAPSHOT/hive-shims-0.13.0-SNAPSHOT.jar
[INFO] Installing /data/hive-ptest/working/apache-svn-trunk-source/shims/aggregator/pom.xml to /data/hive-ptest/working/maven/org/apache/hive/hive-shims/0.13.0-SNAPSHOT/hive-shims-0.13.0-SNAPSHOT.pom
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building Hive Common 0.13.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-common ---
[INFO] Deleting /data/hive-ptest/working/apache-svn-trunk-source/common (includes = [datanucleus.log, derby.log], excludes = [])
[INFO] 
[INFO] --- maven-antrun-plugin:1.7:run (generate-version-annotation) @ hive-common ---
[INFO] Executing tasks

main:
[INFO] Executed tasks
[INFO] 
[INFO] --- build-helper-maven-plugin:1.8:add-source (add-source) @ hive-common ---
[INFO] Source directory: /data/hive-ptest/working/apache-svn-trunk-source/common/src/gen added.
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ hive-common ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO] 
[INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @ hive-common ---
[INFO] Executing tasks

main:
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hive-common ---
[INFO] Compiling 38 source files to /data/hive-ptest/working/apache-svn-trunk-source/common/target/classes
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR : 
[INFO] -------------------------------------------------------------
[ERROR] /data/hive-ptest/working/apache-svn-trunk-source/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[1857,5] invalid method declaration; return type required
[ERROR] /data/hive-ptest/working/apache-svn-trunk-source/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[1857,39] illegal start of type
[ERROR] /data/hive-ptest/working/apache-svn-trunk-source/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[1857,76] illegal start of type
[ERROR] /data/hive-ptest/working/apache-svn-trunk-source/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[1857,82] ';' expected
[INFO] 4 errors 
[INFO] -------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Hive .............................................. SUCCESS [4.656s]
[INFO] Hive Ant Utilities ................................ SUCCESS [10.872s]
[INFO] Hive Shims Common ................................. SUCCESS [4.671s]
[INFO] Hive Shims 0.20 ................................... SUCCESS [2.808s]
[INFO] Hive Shims Secure Common .......................... SUCCESS [3.859s]
[INFO] Hive Shims 0.20S .................................. SUCCESS [1.806s]
[INFO] Hive Shims 0.23 ................................... SUCCESS [5.922s]
[INFO] Hive Shims ........................................ SUCCESS [0.531s]
[INFO] Hive Common ....................................... FAILURE [7.598s]
[INFO] Hive Serde ........................................ SKIPPED
[INFO] Hive Metastore .................................... SKIPPED
[INFO] Hive Query Language ............................... SKIPPED
[INFO] Hive Service ...................................... SKIPPED
[INFO] Hive JDBC ......................................... SKIPPED
[INFO] Hive Beeline ...................................... SKIPPED
[INFO] Hive CLI .......................................... SKIPPED
[INFO] Hive Contrib ...................................... SKIPPED
[INFO] Hive HBase Handler ................................ SKIPPED
[INFO] Hive HCatalog ..................................... SKIPPED
[INFO] Hive HCatalog Core ................................ SKIPPED
[INFO] Hive HCatalog Pig Adapter ......................... SKIPPED
[INFO] Hive HCatalog Server Extensions ................... SKIPPED
[INFO] Hive HCatalog Webhcat Java Client ................. SKIPPED
[INFO] Hive HCatalog Webhcat ............................. SKIPPED
[INFO] Hive HCatalog HBase Storage Handler ............... SKIPPED
[INFO] Hive HWI .......................................... SKIPPED
[INFO] Hive ODBC ......................................... SKIPPED
[INFO] Hive Shims Aggregator ............................. SKIPPED
[INFO] Hive TestUtils .................................... SKIPPED
[INFO] Hive Packaging .................................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 45.887s
[INFO] Finished at: Tue Feb 18 10:48:11 EST 2014
[INFO] Final Memory: 29M/71M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project hive-common: Compilation failure: Compilation failure:
[ERROR] /data/hive-ptest/working/apache-svn-trunk-source/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[1857,5] invalid method declaration; return type required
[ERROR] /data/hive-ptest/working/apache-svn-trunk-source/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[1857,39] illegal start of type
[ERROR] /data/hive-ptest/working/apache-svn-trunk-source/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[1857,76] illegal start of type
[ERROR] /data/hive-ptest/working/apache-svn-trunk-source/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[1857,82] ';' expected
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :hive-common
+ exit 1
'

              This message is automatically generated.

              ATTACHMENT ID: 12629519

              Show
              Hive QA added a comment - Overall : -1 no tests executed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12629519/HIVE-5217.5.patch Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1383/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1383/console Messages: **** This message was trimmed, see log for full details **** [INFO] --- maven-install-plugin:2.4:install (default-install) @ hive-shims-0.20S --- [INFO] Installing /data/hive-ptest/working/apache-svn-trunk-source/shims/0.20S/target/hive-shims-0.20S-0.13.0-SNAPSHOT.jar to /data/hive-ptest/working/maven/org/apache/hive/shims/hive-shims-0.20S/0.13.0-SNAPSHOT/hive-shims-0.20S-0.13.0-SNAPSHOT.jar [INFO] Installing /data/hive-ptest/working/apache-svn-trunk-source/shims/0.20S/pom.xml to /data/hive-ptest/working/maven/org/apache/hive/shims/hive-shims-0.20S/0.13.0-SNAPSHOT/hive-shims-0.20S-0.13.0-SNAPSHOT.pom [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building Hive Shims 0.23 0.13.0-SNAPSHOT [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-shims-0.23 --- [INFO] Deleting /data/hive-ptest/working/apache-svn-trunk-source/shims/0.23 (includes = [datanucleus.log, derby.log], excludes = []) [INFO] [INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ hive-shims-0.23 --- [debug] execute contextualize [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /data/hive-ptest/working/apache-svn-trunk-source/shims/0.23/src/main/resources [INFO] [INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @ hive-shims-0.23 --- [INFO] Executing tasks main: [INFO] Executed tasks [INFO] [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hive-shims-0.23 --- [INFO] Compiling 3 source files to /data/hive-ptest/working/apache-svn-trunk-source/shims/0.23/target/classes [WARNING] Note: /data/hive-ptest/working/apache-svn-trunk-source/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java uses or overrides a deprecated API. [WARNING] Note: Recompile with -Xlint:deprecation for details. [INFO] [INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ hive-shims-0.23 --- [debug] execute contextualize [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /data/hive-ptest/working/apache-svn-trunk-source/shims/0.23/src/test/resources [INFO] [INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ hive-shims-0.23 --- [INFO] Executing tasks main: [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/shims/0.23/target/tmp [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/shims/0.23/target/warehouse [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/shims/0.23/target/tmp/conf [copy] Copying 5 files to /data/hive-ptest/working/apache-svn-trunk-source/shims/0.23/target/tmp/conf [INFO] Executed tasks [INFO] [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ hive-shims-0.23 --- [INFO] No sources to compile [INFO] [INFO] --- maven-surefire-plugin:2.16:test (default-test) @ hive-shims-0.23 --- [INFO] Tests are skipped. [INFO] [INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ hive-shims-0.23 --- [INFO] Building jar: /data/hive-ptest/working/apache-svn-trunk-source/shims/0.23/target/hive-shims-0.23-0.13.0-SNAPSHOT.jar [INFO] [INFO] --- maven-install-plugin:2.4:install (default-install) @ hive-shims-0.23 --- [INFO] Installing /data/hive-ptest/working/apache-svn-trunk-source/shims/0.23/target/hive-shims-0.23-0.13.0-SNAPSHOT.jar to /data/hive-ptest/working/maven/org/apache/hive/shims/hive-shims-0.23/0.13.0-SNAPSHOT/hive-shims-0.23-0.13.0-SNAPSHOT.jar [INFO] Installing /data/hive-ptest/working/apache-svn-trunk-source/shims/0.23/pom.xml to /data/hive-ptest/working/maven/org/apache/hive/shims/hive-shims-0.23/0.13.0-SNAPSHOT/hive-shims-0.23-0.13.0-SNAPSHOT.pom [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building Hive Shims 0.13.0-SNAPSHOT [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-shims --- [INFO] Deleting /data/hive-ptest/working/apache-svn-trunk-source/shims/aggregator (includes = [datanucleus.log, derby.log], excludes = []) [INFO] [INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ hive-shims --- [debug] execute contextualize [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /data/hive-ptest/working/apache-svn-trunk-source/shims/aggregator/src/main/resources [INFO] [INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @ hive-shims --- [INFO] Executing tasks main: [INFO] Executed tasks [INFO] [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hive-shims --- [INFO] No sources to compile [INFO] [INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ hive-shims --- [debug] execute contextualize [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /data/hive-ptest/working/apache-svn-trunk-source/shims/aggregator/src/test/resources [INFO] [INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ hive-shims --- [INFO] Executing tasks main: [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/shims/aggregator/target/tmp [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/shims/aggregator/target/warehouse [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/shims/aggregator/target/tmp/conf [copy] Copying 5 files to /data/hive-ptest/working/apache-svn-trunk-source/shims/aggregator/target/tmp/conf [INFO] Executed tasks [INFO] [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ hive-shims --- [INFO] No sources to compile [INFO] [INFO] --- maven-surefire-plugin:2.16:test (default-test) @ hive-shims --- [INFO] Tests are skipped. [INFO] [INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ hive-shims --- [WARNING] JAR will be empty - no content was marked for inclusion! [INFO] Building jar: /data/hive-ptest/working/apache-svn-trunk-source/shims/aggregator/target/hive-shims-0.13.0-SNAPSHOT.jar [INFO] [INFO] --- maven-install-plugin:2.4:install (default-install) @ hive-shims --- [INFO] Installing /data/hive-ptest/working/apache-svn-trunk-source/shims/aggregator/target/hive-shims-0.13.0-SNAPSHOT.jar to /data/hive-ptest/working/maven/org/apache/hive/hive-shims/0.13.0-SNAPSHOT/hive-shims-0.13.0-SNAPSHOT.jar [INFO] Installing /data/hive-ptest/working/apache-svn-trunk-source/shims/aggregator/pom.xml to /data/hive-ptest/working/maven/org/apache/hive/hive-shims/0.13.0-SNAPSHOT/hive-shims-0.13.0-SNAPSHOT.pom [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building Hive Common 0.13.0-SNAPSHOT [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-common --- [INFO] Deleting /data/hive-ptest/working/apache-svn-trunk-source/common (includes = [datanucleus.log, derby.log], excludes = []) [INFO] [INFO] --- maven-antrun-plugin:1.7:run (generate-version-annotation) @ hive-common --- [INFO] Executing tasks main: [INFO] Executed tasks [INFO] [INFO] --- build-helper-maven-plugin:1.8:add-source (add-source) @ hive-common --- [INFO] Source directory: /data/hive-ptest/working/apache-svn-trunk-source/common/src/gen added. [INFO] [INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ hive-common --- [debug] execute contextualize [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 1 resource [INFO] [INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @ hive-common --- [INFO] Executing tasks main: [INFO] Executed tasks [INFO] [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hive-common --- [INFO] Compiling 38 source files to /data/hive-ptest/working/apache-svn-trunk-source/common/target/classes [INFO] ------------------------------------------------------------- [ERROR] COMPILATION ERROR : [INFO] ------------------------------------------------------------- [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[1857,5] invalid method declaration; return type required [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[1857,39] illegal start of type [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[1857,76] illegal start of type [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[1857,82] ';' expected [INFO] 4 errors [INFO] ------------------------------------------------------------- [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Hive .............................................. SUCCESS [4.656s] [INFO] Hive Ant Utilities ................................ SUCCESS [10.872s] [INFO] Hive Shims Common ................................. SUCCESS [4.671s] [INFO] Hive Shims 0.20 ................................... SUCCESS [2.808s] [INFO] Hive Shims Secure Common .......................... SUCCESS [3.859s] [INFO] Hive Shims 0.20S .................................. SUCCESS [1.806s] [INFO] Hive Shims 0.23 ................................... SUCCESS [5.922s] [INFO] Hive Shims ........................................ SUCCESS [0.531s] [INFO] Hive Common ....................................... FAILURE [7.598s] [INFO] Hive Serde ........................................ SKIPPED [INFO] Hive Metastore .................................... SKIPPED [INFO] Hive Query Language ............................... SKIPPED [INFO] Hive Service ...................................... SKIPPED [INFO] Hive JDBC ......................................... SKIPPED [INFO] Hive Beeline ...................................... SKIPPED [INFO] Hive CLI .......................................... SKIPPED [INFO] Hive Contrib ...................................... SKIPPED [INFO] Hive HBase Handler ................................ SKIPPED [INFO] Hive HCatalog ..................................... SKIPPED [INFO] Hive HCatalog Core ................................ SKIPPED [INFO] Hive HCatalog Pig Adapter ......................... SKIPPED [INFO] Hive HCatalog Server Extensions ................... SKIPPED [INFO] Hive HCatalog Webhcat Java Client ................. SKIPPED [INFO] Hive HCatalog Webhcat ............................. SKIPPED [INFO] Hive HCatalog HBase Storage Handler ............... SKIPPED [INFO] Hive HWI .......................................... SKIPPED [INFO] Hive ODBC ......................................... SKIPPED [INFO] Hive Shims Aggregator ............................. SKIPPED [INFO] Hive TestUtils .................................... SKIPPED [INFO] Hive Packaging .................................... SKIPPED [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 45.887s [INFO] Finished at: Tue Feb 18 10:48:11 EST 2014 [INFO] Final Memory: 29M/71M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project hive-common: Compilation failure: Compilation failure: [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[1857,5] invalid method declaration; return type required [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[1857,39] illegal start of type [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[1857,76] illegal start of type [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[1857,82] ';' expected [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn <goals> -rf :hive-common + exit 1 ' This message is automatically generated. ATTACHMENT ID: 12629519
              Hide
              Permalink
              Vaibhav Gumashta added a comment -

              Carl Steinbach Thejas M Nair I just uploaded a patch which moves the polling timeout logic to CLIService. It would be awesome if you can take a look. Thanks in advance!

              Show
              Vaibhav Gumashta added a comment - Carl Steinbach Thejas M Nair I just uploaded a patch which moves the polling timeout logic to CLIService. It would be awesome if you can take a look. Thanks in advance!
              Hide
              Permalink
              Vaibhav Gumashta added a comment -

              Carl Steinbach Thejas M Nair Any further thoughts on the discussion so far? Thanks!

              Show
              Vaibhav Gumashta added a comment - Carl Steinbach Thejas M Nair Any further thoughts on the discussion so far? Thanks!
              Hide
              Permalink
              Vaibhav Gumashta added a comment -

              Carl Steinbach Look forward to your feedback on this one. Thanks!

              Show
              Vaibhav Gumashta added a comment - Carl Steinbach Look forward to your feedback on this one. Thanks!
              Hide
              Permalink
              Vaibhav Gumashta added a comment -

              Carl Steinbach Uploaded a patch based on the feedback.

              Regarding moving the long polling timeout logic, I have some comments where I'll appreciate your feedback:
              1. Moving that logic to CLIService would mean accessing some of the operation logic within CLIService, and from the general design of CLIService it looks like we wanted to avoid that.
              2. This will also mean that the async nature moves to Operation (runAsync, backgroundHandle) from SQLOperation.

              What are your thoughts? Thanks for the feedback and discussion so far!

              Show
              Vaibhav Gumashta added a comment - Carl Steinbach Uploaded a patch based on the feedback. Regarding moving the long polling timeout logic, I have some comments where I'll appreciate your feedback: 1. Moving that logic to CLIService would mean accessing some of the operation logic within CLIService, and from the general design of CLIService it looks like we wanted to avoid that. 2. This will also mean that the async nature moves to Operation (runAsync, backgroundHandle) from SQLOperation. What are your thoughts? Thanks for the feedback and discussion so far!
              Hide
              Permalink
              Carl Steinbach added a comment -

              I left some more comments on reviewboard.

              Are you planning to move the long polling timeout logic from SQLOperation.getState() to CLIService?

              Show
              Carl Steinbach added a comment - I left some more comments on reviewboard. Are you planning to move the long polling timeout logic from SQLOperation.getState() to CLIService?
              Hide
              Permalink
              Vaibhav Gumashta added a comment -

              Carl Steinbach Yes, you can take a look now. Thanks.

              Show
              Vaibhav Gumashta added a comment - Carl Steinbach Yes, you can take a look now. Thanks.
              Hide
              Permalink
              Carl Steinbach added a comment -

              [~vaibhavgumashta] Is the patch ready for review?

              Show
              Carl Steinbach added a comment - [~vaibhavgumashta] Is the patch ready for review?
              Hide
              Permalink
              Vaibhav Gumashta added a comment -

              Couldn't update the other rb link as I wasn't the creator. Link for latest changes: https://reviews.apache.org/r/15435/

              Show
              Vaibhav Gumashta added a comment - Couldn't update the other rb link as I wasn't the creator. Link for latest changes: https://reviews.apache.org/r/15435/
              Hide
              Permalink
              Vaibhav Gumashta added a comment -

              Thanks Carl Steinbach. I'll look the the feedback here and on rb.

              Meanwhile on your last comment: getOperationStatus returns after hive.server2.long.polling.timeout expires or the query completes - whichever happens first. So if polling.timeout = 5000 and the query finishes in 1000, the call to getOperationStatus will return in 1000. However, in the test case, the test query we are running is very short - therefore to test the behavior of the long polling timeout, we're explicitly setting the config to a smaller time than default, which is 5000; otherwise if the query takes 3000, it will return state FINISHED and we won't be able to test the timeout.

              Show
              Vaibhav Gumashta added a comment - Thanks Carl Steinbach . I'll look the the feedback here and on rb. Meanwhile on your last comment: getOperationStatus returns after hive.server2.long.polling.timeout expires or the query completes - whichever happens first. So if polling.timeout = 5000 and the query finishes in 1000, the call to getOperationStatus will return in 1000. However, in the test case, the test query we are running is very short - therefore to test the behavior of the long polling timeout, we're explicitly setting the config to a smaller time than default, which is 5000; otherwise if the query takes 3000, it will return state FINISHED and we won't be able to test the timeout.
              Hide
              Permalink
              Carl Steinbach added a comment -

              I posted the patch to reviewboard and left some comments: https://reviews.apache.org/r/15337/

              Based on the test case included with the patch it looks like hive.server2.long.polling.timeout sets a lower bound on the total round trip time required to execute a getOperationStatus RPC, e.g. if polling.timeout = 5000 my query can finish after a second but getOperationStatus will still block for another four seconds before returning. Is this accurate?

              Show
              Carl Steinbach added a comment - I posted the patch to reviewboard and left some comments: https://reviews.apache.org/r/15337/ Based on the test case included with the patch it looks like hive.server2.long.polling.timeout sets a lower bound on the total round trip time required to execute a getOperationStatus RPC, e.g. if polling.timeout = 5000 my query can finish after a second but getOperationStatus will still block for another four seconds before returning. Is this accurate?
              Hide
              Permalink
              Carl Steinbach added a comment -

              It wouldn't be too hard to add support for a non long polling getOperationStatus call in future, specially a non rpc call.

              Adding a getOperationStatusInternal() or getOperationStatusNonLongPoll() method to work around this problem is exactly the sort of thing I want to avoid. This patch implements a service layer feature, so I think it makes sense that the implementation belongs in the service layer as well. In CLIService.getOperationStatus() you have access to the SessionManager, and from that you can easily get the Session object, the Operation object, the operation type, and the Session's configuration. As an added bonus you also get the OperationHandle which makes it easier to emit useful log messages that reference the session and operation IDs.

              Show
              Carl Steinbach added a comment - It wouldn't be too hard to add support for a non long polling getOperationStatus call in future, specially a non rpc call. Adding a getOperationStatusInternal() or getOperationStatusNonLongPoll() method to work around this problem is exactly the sort of thing I want to avoid. This patch implements a service layer feature, so I think it makes sense that the implementation belongs in the service layer as well. In CLIService.getOperationStatus() you have access to the SessionManager, and from that you can easily get the Session object, the Operation object, the operation type, and the Session's configuration. As an added bonus you also get the OperationHandle which makes it easier to emit useful log messages that reference the session and operation IDs.
              Hide
              Permalink
              Thejas M Nair added a comment -

              Carl Steinbach What Vaibhav is saying sounds reasonable to me. It wouldn't be too hard to add support for a non long polling getOperationStatus call in future, specially a non rpc call. We can make the change when we have such a use case.

              Show
              Thejas M Nair added a comment - Carl Steinbach What Vaibhav is saying sounds reasonable to me. It wouldn't be too hard to add support for a non long polling getOperationStatus call in future, specially a non rpc call. We can make the change when we have such a use case.
              Hide
              Permalink
              Hive QA added a comment -

              Overall: +1 all checks pass

              Here are the results of testing the latest attachment:
              https://issues.apache.org/jira/secure/attachment/12612471/HIVE-5217.D12801.6.patch

              SUCCESS: +1 4577 tests passed

              Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/168/testReport
              Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/168/console

              Messages:

              Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase

              This message is automatically generated.

              ATTACHMENT ID: 12612471

              Show
              Hive QA added a comment - Overall : +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12612471/HIVE-5217.D12801.6.patch SUCCESS: +1 4577 tests passed Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/168/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/168/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated. ATTACHMENT ID: 12612471
              Hide
              Permalink
              Vaibhav Gumashta added a comment -

              Patch rebased on trunk

              Show
              Vaibhav Gumashta added a comment - Patch rebased on trunk
              Hide
              Permalink
              Vaibhav Gumashta added a comment -

              Carl Steinbach Sure, let me rebase on the trunk. You can look at the patch here: https://reviews.facebook.net/D12801, but I can create a request on RB if you prefer that.

              Actually the delay period only affects the async SQLOperation (and returns immediately if the async call has the results) and doesn't change the blocking calls. Also, as far as I understand the use case you are pointing to is for a cleanup type of operation, where we might want to clean up the entire session based on whether the client is active or not. In that case, the cleanup thread might end up checking the last active time of the session rather than the operation state and based on whether it exceeds some threshold, delete the session from the server.

              Moving the long polling logic to CLIService might involve changing the Operation API, which I was trying to avoid until we get to a specific use case. But if you feel otherwise, I can definitely make the change.

              Thanks for looking at the patch!

              Show
              Vaibhav Gumashta added a comment - Carl Steinbach Sure, let me rebase on the trunk. You can look at the patch here: https://reviews.facebook.net/D12801 , but I can create a request on RB if you prefer that. Actually the delay period only affects the async SQLOperation (and returns immediately if the async call has the results) and doesn't change the blocking calls. Also, as far as I understand the use case you are pointing to is for a cleanup type of operation, where we might want to clean up the entire session based on whether the client is active or not. In that case, the cleanup thread might end up checking the last active time of the session rather than the operation state and based on whether it exceeds some threshold, delete the session from the server. Moving the long polling logic to CLIService might involve changing the Operation API, which I was trying to avoid until we get to a specific use case. But if you feel otherwise, I can definitely make the change. Thanks for looking at the patch!
              Hide
              Permalink
              Carl Steinbach added a comment -

              [~vaibhavgumashta] Version 5 of the patch won't apply cleanly on trunk. Can you please rebase the patch and post a review request on RB?

              One concern I have with the patch is that the delay period affects everyone who calls SQLOperation.getStatus(). Ideally we would be able to limit this behavior to remote clients only. I'm worried that at some point in the future we're going to want to have a housekeeping or monitoring thread that periodically calls getStatus() on all active operations, and we don't want getStatus() to block in these types of situations. Would it be practical to relocate this logic to the CLIService layer so that it only impacts CLIService clients?

              Show
              Carl Steinbach added a comment - [~vaibhavgumashta] Version 5 of the patch won't apply cleanly on trunk. Can you please rebase the patch and post a review request on RB? One concern I have with the patch is that the delay period affects everyone who calls SQLOperation.getStatus(). Ideally we would be able to limit this behavior to remote clients only. I'm worried that at some point in the future we're going to want to have a housekeeping or monitoring thread that periodically calls getStatus() on all active operations, and we don't want getStatus() to block in these types of situations. Would it be practical to relocate this logic to the CLIService layer so that it only impacts CLIService clients?
              Hide
              Permalink
              Vaibhav Gumashta added a comment -

              Sorry missed out attaching the patch. Thanks for the review!

              Show
              Vaibhav Gumashta added a comment - Sorry missed out attaching the patch. Thanks for the review!
              Hide
              Permalink
              Thejas M Nair added a comment -

              +1 for latest patch in phabricator. Can you please upload that patch to jira as well ?

              Show
              Thejas M Nair added a comment - +1 for latest patch in phabricator. Can you please upload that patch to jira as well ?
              Hide
              Permalink
              Vaibhav Gumashta added a comment -

              Phab link: https://reviews.facebook.net/D12801

              Show
              Vaibhav Gumashta added a comment - Phab link: https://reviews.facebook.net/D12801
              Hide
              Permalink
              Vaibhav Gumashta added a comment -

              Here is the idea: http://tools.ietf.org/html/draft-loreto-http-bidirectional-07.

              Show
              Vaibhav Gumashta added a comment - Here is the idea: http://tools.ietf.org/html/draft-loreto-http-bidirectional-07 .

                People

                • Assignee:
                  Vaibhav Gumashta
                  Reporter:
                  Vaibhav Gumashta
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  6 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Development