Details

    • Type: Sub-task Sub-task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.13.0
    • Fix Version/s: 0.13.0
    • Component/s: HiveServer2
    • Labels:
      None
    • Release Note:
      Hide
      Change to use Long polling as described in description.
      Adds hive.server2.long.polling.timeout configuration parameter, which can be used to configure how long the long poll waits. Most users would not need to bother about changing this configuration parameter.
      Show
      Change to use Long polling as described in description. Adds hive.server2.long.polling.timeout configuration parameter, which can be used to configure how long the long poll waits. Most users would not need to bother about changing this configuration parameter.

      Description

      HIVE-4617 provides support for async execution in HS2. The client gets an operation handle which it can poll to check on the operation status. However, the polling frequency is entirely left to the client which can be resource inefficient. Long polling will solve this, by blocking the client request to check the operation status for a configurable amount of time (a new HS2 config) if the data is not available, but responding immediately if the data is available.

      1. HIVE-5217.D12801.6.patch
        8 kB
        Vaibhav Gumashta
      2. HIVE-5217.D12801.5.patch
        8 kB
        Vaibhav Gumashta
      3. HIVE-5217.D12801.4.patch
        8 kB
        Vaibhav Gumashta
      4. HIVE-5217.D12801.3.patch
        5 kB
        Vaibhav Gumashta
      5. HIVE-5217.D12801.2.patch
        5 kB
        Vaibhav Gumashta
      6. HIVE-5217.6.patch
        28 kB
        Vaibhav Gumashta
      7. HIVE-5217.5.patch
        25 kB
        Vaibhav Gumashta
      8. HIVE-5217.4.patch
        27 kB
        Vaibhav Gumashta
      9. HIVE-5217.3.patch
        17 kB
        Vaibhav Gumashta
      10. HIVE-5217.2.patch
        19 kB
        Vaibhav Gumashta

        Issue Links

          Activity

          Hide
          Lefty Leverenz added a comment -

          hive.server2.long.polling.timeout is documented in the wiki here:

          Show
          Lefty Leverenz added a comment - hive.server2.long.polling.timeout is documented in the wiki here: Configuration Properties: hive.server2.long.polling.timeout
          Hide
          Thejas M Nair added a comment -

          Patch committed to trunk.
          Thanks for the contribution Vaibhav!

          Can you add see if the release note looks OK ? Please feel free to edit it.

          Show
          Thejas M Nair added a comment - Patch committed to trunk. Thanks for the contribution Vaibhav! Can you add see if the release note looks OK ? Please feel free to edit it.
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12630053/HIVE-5217.6.patch

          ERROR: -1 due to 2 failed/errored test(s), 5173 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
          org.apache.hcatalog.hbase.TestHBaseDirectOutputFormat.org.apache.hcatalog.hbase.TestHBaseDirectOutputFormat
          

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1439/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1439/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 2 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12630053

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12630053/HIVE-5217.6.patch ERROR: -1 due to 2 failed/errored test(s), 5173 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table org.apache.hcatalog.hbase.TestHBaseDirectOutputFormat.org.apache.hcatalog.hbase.TestHBaseDirectOutputFormat Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1439/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1439/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed This message is automatically generated. ATTACHMENT ID: 12630053
          Hide
          Thejas M Nair added a comment -

          +1

          Show
          Thejas M Nair added a comment - +1
          Hide
          Vaibhav Gumashta added a comment -

          Incorporating Thejas' feedback given on RB.

          Show
          Vaibhav Gumashta added a comment - Incorporating Thejas' feedback given on RB.
          Hide
          Hive QA added a comment -

          Overall: -1 no tests executed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12629519/HIVE-5217.5.patch

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1383/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1383/console

          Messages:

          **** This message was trimmed, see log for full details ****
          [INFO] --- maven-install-plugin:2.4:install (default-install) @ hive-shims-0.20S ---
          [INFO] Installing /data/hive-ptest/working/apache-svn-trunk-source/shims/0.20S/target/hive-shims-0.20S-0.13.0-SNAPSHOT.jar to /data/hive-ptest/working/maven/org/apache/hive/shims/hive-shims-0.20S/0.13.0-SNAPSHOT/hive-shims-0.20S-0.13.0-SNAPSHOT.jar
          [INFO] Installing /data/hive-ptest/working/apache-svn-trunk-source/shims/0.20S/pom.xml to /data/hive-ptest/working/maven/org/apache/hive/shims/hive-shims-0.20S/0.13.0-SNAPSHOT/hive-shims-0.20S-0.13.0-SNAPSHOT.pom
          [INFO]                                                                         
          [INFO] ------------------------------------------------------------------------
          [INFO] Building Hive Shims 0.23 0.13.0-SNAPSHOT
          [INFO] ------------------------------------------------------------------------
          [INFO] 
          [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-shims-0.23 ---
          [INFO] Deleting /data/hive-ptest/working/apache-svn-trunk-source/shims/0.23 (includes = [datanucleus.log, derby.log], excludes = [])
          [INFO] 
          [INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ hive-shims-0.23 ---
          [debug] execute contextualize
          [INFO] Using 'UTF-8' encoding to copy filtered resources.
          [INFO] skip non existing resourceDirectory /data/hive-ptest/working/apache-svn-trunk-source/shims/0.23/src/main/resources
          [INFO] 
          [INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @ hive-shims-0.23 ---
          [INFO] Executing tasks
          
          main:
          [INFO] Executed tasks
          [INFO] 
          [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hive-shims-0.23 ---
          [INFO] Compiling 3 source files to /data/hive-ptest/working/apache-svn-trunk-source/shims/0.23/target/classes
          [WARNING] Note: /data/hive-ptest/working/apache-svn-trunk-source/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java uses or overrides a deprecated API.
          [WARNING] Note: Recompile with -Xlint:deprecation for details.
          [INFO] 
          [INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ hive-shims-0.23 ---
          [debug] execute contextualize
          [INFO] Using 'UTF-8' encoding to copy filtered resources.
          [INFO] skip non existing resourceDirectory /data/hive-ptest/working/apache-svn-trunk-source/shims/0.23/src/test/resources
          [INFO] 
          [INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ hive-shims-0.23 ---
          [INFO] Executing tasks
          
          main:
              [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/shims/0.23/target/tmp
              [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/shims/0.23/target/warehouse
              [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/shims/0.23/target/tmp/conf
               [copy] Copying 5 files to /data/hive-ptest/working/apache-svn-trunk-source/shims/0.23/target/tmp/conf
          [INFO] Executed tasks
          [INFO] 
          [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ hive-shims-0.23 ---
          [INFO] No sources to compile
          [INFO] 
          [INFO] --- maven-surefire-plugin:2.16:test (default-test) @ hive-shims-0.23 ---
          [INFO] Tests are skipped.
          [INFO] 
          [INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ hive-shims-0.23 ---
          [INFO] Building jar: /data/hive-ptest/working/apache-svn-trunk-source/shims/0.23/target/hive-shims-0.23-0.13.0-SNAPSHOT.jar
          [INFO] 
          [INFO] --- maven-install-plugin:2.4:install (default-install) @ hive-shims-0.23 ---
          [INFO] Installing /data/hive-ptest/working/apache-svn-trunk-source/shims/0.23/target/hive-shims-0.23-0.13.0-SNAPSHOT.jar to /data/hive-ptest/working/maven/org/apache/hive/shims/hive-shims-0.23/0.13.0-SNAPSHOT/hive-shims-0.23-0.13.0-SNAPSHOT.jar
          [INFO] Installing /data/hive-ptest/working/apache-svn-trunk-source/shims/0.23/pom.xml to /data/hive-ptest/working/maven/org/apache/hive/shims/hive-shims-0.23/0.13.0-SNAPSHOT/hive-shims-0.23-0.13.0-SNAPSHOT.pom
          [INFO]                                                                         
          [INFO] ------------------------------------------------------------------------
          [INFO] Building Hive Shims 0.13.0-SNAPSHOT
          [INFO] ------------------------------------------------------------------------
          [INFO] 
          [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-shims ---
          [INFO] Deleting /data/hive-ptest/working/apache-svn-trunk-source/shims/aggregator (includes = [datanucleus.log, derby.log], excludes = [])
          [INFO] 
          [INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ hive-shims ---
          [debug] execute contextualize
          [INFO] Using 'UTF-8' encoding to copy filtered resources.
          [INFO] skip non existing resourceDirectory /data/hive-ptest/working/apache-svn-trunk-source/shims/aggregator/src/main/resources
          [INFO] 
          [INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @ hive-shims ---
          [INFO] Executing tasks
          
          main:
          [INFO] Executed tasks
          [INFO] 
          [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hive-shims ---
          [INFO] No sources to compile
          [INFO] 
          [INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ hive-shims ---
          [debug] execute contextualize
          [INFO] Using 'UTF-8' encoding to copy filtered resources.
          [INFO] skip non existing resourceDirectory /data/hive-ptest/working/apache-svn-trunk-source/shims/aggregator/src/test/resources
          [INFO] 
          [INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ hive-shims ---
          [INFO] Executing tasks
          
          main:
              [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/shims/aggregator/target/tmp
              [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/shims/aggregator/target/warehouse
              [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/shims/aggregator/target/tmp/conf
               [copy] Copying 5 files to /data/hive-ptest/working/apache-svn-trunk-source/shims/aggregator/target/tmp/conf
          [INFO] Executed tasks
          [INFO] 
          [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ hive-shims ---
          [INFO] No sources to compile
          [INFO] 
          [INFO] --- maven-surefire-plugin:2.16:test (default-test) @ hive-shims ---
          [INFO] Tests are skipped.
          [INFO] 
          [INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ hive-shims ---
          [WARNING] JAR will be empty - no content was marked for inclusion!
          [INFO] Building jar: /data/hive-ptest/working/apache-svn-trunk-source/shims/aggregator/target/hive-shims-0.13.0-SNAPSHOT.jar
          [INFO] 
          [INFO] --- maven-install-plugin:2.4:install (default-install) @ hive-shims ---
          [INFO] Installing /data/hive-ptest/working/apache-svn-trunk-source/shims/aggregator/target/hive-shims-0.13.0-SNAPSHOT.jar to /data/hive-ptest/working/maven/org/apache/hive/hive-shims/0.13.0-SNAPSHOT/hive-shims-0.13.0-SNAPSHOT.jar
          [INFO] Installing /data/hive-ptest/working/apache-svn-trunk-source/shims/aggregator/pom.xml to /data/hive-ptest/working/maven/org/apache/hive/hive-shims/0.13.0-SNAPSHOT/hive-shims-0.13.0-SNAPSHOT.pom
          [INFO]                                                                         
          [INFO] ------------------------------------------------------------------------
          [INFO] Building Hive Common 0.13.0-SNAPSHOT
          [INFO] ------------------------------------------------------------------------
          [INFO] 
          [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-common ---
          [INFO] Deleting /data/hive-ptest/working/apache-svn-trunk-source/common (includes = [datanucleus.log, derby.log], excludes = [])
          [INFO] 
          [INFO] --- maven-antrun-plugin:1.7:run (generate-version-annotation) @ hive-common ---
          [INFO] Executing tasks
          
          main:
          [INFO] Executed tasks
          [INFO] 
          [INFO] --- build-helper-maven-plugin:1.8:add-source (add-source) @ hive-common ---
          [INFO] Source directory: /data/hive-ptest/working/apache-svn-trunk-source/common/src/gen added.
          [INFO] 
          [INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ hive-common ---
          [debug] execute contextualize
          [INFO] Using 'UTF-8' encoding to copy filtered resources.
          [INFO] Copying 1 resource
          [INFO] 
          [INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @ hive-common ---
          [INFO] Executing tasks
          
          main:
          [INFO] Executed tasks
          [INFO] 
          [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hive-common ---
          [INFO] Compiling 38 source files to /data/hive-ptest/working/apache-svn-trunk-source/common/target/classes
          [INFO] -------------------------------------------------------------
          [ERROR] COMPILATION ERROR : 
          [INFO] -------------------------------------------------------------
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[1857,5] invalid method declaration; return type required
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[1857,39] illegal start of type
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[1857,76] illegal start of type
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[1857,82] ';' expected
          [INFO] 4 errors 
          [INFO] -------------------------------------------------------------
          [INFO] ------------------------------------------------------------------------
          [INFO] Reactor Summary:
          [INFO] 
          [INFO] Hive .............................................. SUCCESS [4.656s]
          [INFO] Hive Ant Utilities ................................ SUCCESS [10.872s]
          [INFO] Hive Shims Common ................................. SUCCESS [4.671s]
          [INFO] Hive Shims 0.20 ................................... SUCCESS [2.808s]
          [INFO] Hive Shims Secure Common .......................... SUCCESS [3.859s]
          [INFO] Hive Shims 0.20S .................................. SUCCESS [1.806s]
          [INFO] Hive Shims 0.23 ................................... SUCCESS [5.922s]
          [INFO] Hive Shims ........................................ SUCCESS [0.531s]
          [INFO] Hive Common ....................................... FAILURE [7.598s]
          [INFO] Hive Serde ........................................ SKIPPED
          [INFO] Hive Metastore .................................... SKIPPED
          [INFO] Hive Query Language ............................... SKIPPED
          [INFO] Hive Service ...................................... SKIPPED
          [INFO] Hive JDBC ......................................... SKIPPED
          [INFO] Hive Beeline ...................................... SKIPPED
          [INFO] Hive CLI .......................................... SKIPPED
          [INFO] Hive Contrib ...................................... SKIPPED
          [INFO] Hive HBase Handler ................................ SKIPPED
          [INFO] Hive HCatalog ..................................... SKIPPED
          [INFO] Hive HCatalog Core ................................ SKIPPED
          [INFO] Hive HCatalog Pig Adapter ......................... SKIPPED
          [INFO] Hive HCatalog Server Extensions ................... SKIPPED
          [INFO] Hive HCatalog Webhcat Java Client ................. SKIPPED
          [INFO] Hive HCatalog Webhcat ............................. SKIPPED
          [INFO] Hive HCatalog HBase Storage Handler ............... SKIPPED
          [INFO] Hive HWI .......................................... SKIPPED
          [INFO] Hive ODBC ......................................... SKIPPED
          [INFO] Hive Shims Aggregator ............................. SKIPPED
          [INFO] Hive TestUtils .................................... SKIPPED
          [INFO] Hive Packaging .................................... SKIPPED
          [INFO] ------------------------------------------------------------------------
          [INFO] BUILD FAILURE
          [INFO] ------------------------------------------------------------------------
          [INFO] Total time: 45.887s
          [INFO] Finished at: Tue Feb 18 10:48:11 EST 2014
          [INFO] Final Memory: 29M/71M
          [INFO] ------------------------------------------------------------------------
          [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project hive-common: Compilation failure: Compilation failure:
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[1857,5] invalid method declaration; return type required
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[1857,39] illegal start of type
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[1857,76] illegal start of type
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[1857,82] ';' expected
          [ERROR] -> [Help 1]
          [ERROR] 
          [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
          [ERROR] Re-run Maven using the -X switch to enable full debug logging.
          [ERROR] 
          [ERROR] For more information about the errors and possible solutions, please read the following articles:
          [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
          [ERROR] 
          [ERROR] After correcting the problems, you can resume the build with the command
          [ERROR]   mvn <goals> -rf :hive-common
          + exit 1
          '
          

          This message is automatically generated.

          ATTACHMENT ID: 12629519

          Show
          Hive QA added a comment - Overall : -1 no tests executed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12629519/HIVE-5217.5.patch Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1383/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1383/console Messages: **** This message was trimmed, see log for full details **** [INFO] --- maven-install-plugin:2.4:install (default-install) @ hive-shims-0.20S --- [INFO] Installing /data/hive-ptest/working/apache-svn-trunk-source/shims/0.20S/target/hive-shims-0.20S-0.13.0-SNAPSHOT.jar to /data/hive-ptest/working/maven/org/apache/hive/shims/hive-shims-0.20S/0.13.0-SNAPSHOT/hive-shims-0.20S-0.13.0-SNAPSHOT.jar [INFO] Installing /data/hive-ptest/working/apache-svn-trunk-source/shims/0.20S/pom.xml to /data/hive-ptest/working/maven/org/apache/hive/shims/hive-shims-0.20S/0.13.0-SNAPSHOT/hive-shims-0.20S-0.13.0-SNAPSHOT.pom [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building Hive Shims 0.23 0.13.0-SNAPSHOT [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-shims-0.23 --- [INFO] Deleting /data/hive-ptest/working/apache-svn-trunk-source/shims/0.23 (includes = [datanucleus.log, derby.log], excludes = []) [INFO] [INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ hive-shims-0.23 --- [debug] execute contextualize [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /data/hive-ptest/working/apache-svn-trunk-source/shims/0.23/src/main/resources [INFO] [INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @ hive-shims-0.23 --- [INFO] Executing tasks main: [INFO] Executed tasks [INFO] [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hive-shims-0.23 --- [INFO] Compiling 3 source files to /data/hive-ptest/working/apache-svn-trunk-source/shims/0.23/target/classes [WARNING] Note: /data/hive-ptest/working/apache-svn-trunk-source/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java uses or overrides a deprecated API. [WARNING] Note: Recompile with -Xlint:deprecation for details. [INFO] [INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ hive-shims-0.23 --- [debug] execute contextualize [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /data/hive-ptest/working/apache-svn-trunk-source/shims/0.23/src/test/resources [INFO] [INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ hive-shims-0.23 --- [INFO] Executing tasks main: [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/shims/0.23/target/tmp [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/shims/0.23/target/warehouse [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/shims/0.23/target/tmp/conf [copy] Copying 5 files to /data/hive-ptest/working/apache-svn-trunk-source/shims/0.23/target/tmp/conf [INFO] Executed tasks [INFO] [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ hive-shims-0.23 --- [INFO] No sources to compile [INFO] [INFO] --- maven-surefire-plugin:2.16:test (default-test) @ hive-shims-0.23 --- [INFO] Tests are skipped. [INFO] [INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ hive-shims-0.23 --- [INFO] Building jar: /data/hive-ptest/working/apache-svn-trunk-source/shims/0.23/target/hive-shims-0.23-0.13.0-SNAPSHOT.jar [INFO] [INFO] --- maven-install-plugin:2.4:install (default-install) @ hive-shims-0.23 --- [INFO] Installing /data/hive-ptest/working/apache-svn-trunk-source/shims/0.23/target/hive-shims-0.23-0.13.0-SNAPSHOT.jar to /data/hive-ptest/working/maven/org/apache/hive/shims/hive-shims-0.23/0.13.0-SNAPSHOT/hive-shims-0.23-0.13.0-SNAPSHOT.jar [INFO] Installing /data/hive-ptest/working/apache-svn-trunk-source/shims/0.23/pom.xml to /data/hive-ptest/working/maven/org/apache/hive/shims/hive-shims-0.23/0.13.0-SNAPSHOT/hive-shims-0.23-0.13.0-SNAPSHOT.pom [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building Hive Shims 0.13.0-SNAPSHOT [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-shims --- [INFO] Deleting /data/hive-ptest/working/apache-svn-trunk-source/shims/aggregator (includes = [datanucleus.log, derby.log], excludes = []) [INFO] [INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ hive-shims --- [debug] execute contextualize [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /data/hive-ptest/working/apache-svn-trunk-source/shims/aggregator/src/main/resources [INFO] [INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @ hive-shims --- [INFO] Executing tasks main: [INFO] Executed tasks [INFO] [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hive-shims --- [INFO] No sources to compile [INFO] [INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ hive-shims --- [debug] execute contextualize [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /data/hive-ptest/working/apache-svn-trunk-source/shims/aggregator/src/test/resources [INFO] [INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ hive-shims --- [INFO] Executing tasks main: [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/shims/aggregator/target/tmp [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/shims/aggregator/target/warehouse [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/shims/aggregator/target/tmp/conf [copy] Copying 5 files to /data/hive-ptest/working/apache-svn-trunk-source/shims/aggregator/target/tmp/conf [INFO] Executed tasks [INFO] [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ hive-shims --- [INFO] No sources to compile [INFO] [INFO] --- maven-surefire-plugin:2.16:test (default-test) @ hive-shims --- [INFO] Tests are skipped. [INFO] [INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ hive-shims --- [WARNING] JAR will be empty - no content was marked for inclusion! [INFO] Building jar: /data/hive-ptest/working/apache-svn-trunk-source/shims/aggregator/target/hive-shims-0.13.0-SNAPSHOT.jar [INFO] [INFO] --- maven-install-plugin:2.4:install (default-install) @ hive-shims --- [INFO] Installing /data/hive-ptest/working/apache-svn-trunk-source/shims/aggregator/target/hive-shims-0.13.0-SNAPSHOT.jar to /data/hive-ptest/working/maven/org/apache/hive/hive-shims/0.13.0-SNAPSHOT/hive-shims-0.13.0-SNAPSHOT.jar [INFO] Installing /data/hive-ptest/working/apache-svn-trunk-source/shims/aggregator/pom.xml to /data/hive-ptest/working/maven/org/apache/hive/hive-shims/0.13.0-SNAPSHOT/hive-shims-0.13.0-SNAPSHOT.pom [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building Hive Common 0.13.0-SNAPSHOT [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-common --- [INFO] Deleting /data/hive-ptest/working/apache-svn-trunk-source/common (includes = [datanucleus.log, derby.log], excludes = []) [INFO] [INFO] --- maven-antrun-plugin:1.7:run (generate-version-annotation) @ hive-common --- [INFO] Executing tasks main: [INFO] Executed tasks [INFO] [INFO] --- build-helper-maven-plugin:1.8:add-source (add-source) @ hive-common --- [INFO] Source directory: /data/hive-ptest/working/apache-svn-trunk-source/common/src/gen added. [INFO] [INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ hive-common --- [debug] execute contextualize [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 1 resource [INFO] [INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @ hive-common --- [INFO] Executing tasks main: [INFO] Executed tasks [INFO] [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hive-common --- [INFO] Compiling 38 source files to /data/hive-ptest/working/apache-svn-trunk-source/common/target/classes [INFO] ------------------------------------------------------------- [ERROR] COMPILATION ERROR : [INFO] ------------------------------------------------------------- [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[1857,5] invalid method declaration; return type required [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[1857,39] illegal start of type [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[1857,76] illegal start of type [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[1857,82] ';' expected [INFO] 4 errors [INFO] ------------------------------------------------------------- [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Hive .............................................. SUCCESS [4.656s] [INFO] Hive Ant Utilities ................................ SUCCESS [10.872s] [INFO] Hive Shims Common ................................. SUCCESS [4.671s] [INFO] Hive Shims 0.20 ................................... SUCCESS [2.808s] [INFO] Hive Shims Secure Common .......................... SUCCESS [3.859s] [INFO] Hive Shims 0.20S .................................. SUCCESS [1.806s] [INFO] Hive Shims 0.23 ................................... SUCCESS [5.922s] [INFO] Hive Shims ........................................ SUCCESS [0.531s] [INFO] Hive Common ....................................... FAILURE [7.598s] [INFO] Hive Serde ........................................ SKIPPED [INFO] Hive Metastore .................................... SKIPPED [INFO] Hive Query Language ............................... SKIPPED [INFO] Hive Service ...................................... SKIPPED [INFO] Hive JDBC ......................................... SKIPPED [INFO] Hive Beeline ...................................... SKIPPED [INFO] Hive CLI .......................................... SKIPPED [INFO] Hive Contrib ...................................... SKIPPED [INFO] Hive HBase Handler ................................ SKIPPED [INFO] Hive HCatalog ..................................... SKIPPED [INFO] Hive HCatalog Core ................................ SKIPPED [INFO] Hive HCatalog Pig Adapter ......................... SKIPPED [INFO] Hive HCatalog Server Extensions ................... SKIPPED [INFO] Hive HCatalog Webhcat Java Client ................. SKIPPED [INFO] Hive HCatalog Webhcat ............................. SKIPPED [INFO] Hive HCatalog HBase Storage Handler ............... SKIPPED [INFO] Hive HWI .......................................... SKIPPED [INFO] Hive ODBC ......................................... SKIPPED [INFO] Hive Shims Aggregator ............................. SKIPPED [INFO] Hive TestUtils .................................... SKIPPED [INFO] Hive Packaging .................................... SKIPPED [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 45.887s [INFO] Finished at: Tue Feb 18 10:48:11 EST 2014 [INFO] Final Memory: 29M/71M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project hive-common: Compilation failure: Compilation failure: [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[1857,5] invalid method declaration; return type required [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[1857,39] illegal start of type [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[1857,76] illegal start of type [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:[1857,82] ';' expected [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn <goals> -rf :hive-common + exit 1 ' This message is automatically generated. ATTACHMENT ID: 12629519
          Hide
          Vaibhav Gumashta added a comment -

          Carl Steinbach Thejas M Nair I just uploaded a patch which moves the polling timeout logic to CLIService. It would be awesome if you can take a look. Thanks in advance!

          Show
          Vaibhav Gumashta added a comment - Carl Steinbach Thejas M Nair I just uploaded a patch which moves the polling timeout logic to CLIService. It would be awesome if you can take a look. Thanks in advance!
          Hide
          Vaibhav Gumashta added a comment -

          Carl Steinbach Thejas M Nair Any further thoughts on the discussion so far? Thanks!

          Show
          Vaibhav Gumashta added a comment - Carl Steinbach Thejas M Nair Any further thoughts on the discussion so far? Thanks!
          Hide
          Vaibhav Gumashta added a comment -

          Carl Steinbach Look forward to your feedback on this one. Thanks!

          Show
          Vaibhav Gumashta added a comment - Carl Steinbach Look forward to your feedback on this one. Thanks!
          Hide
          Vaibhav Gumashta added a comment -

          Carl Steinbach Uploaded a patch based on the feedback.

          Regarding moving the long polling timeout logic, I have some comments where I'll appreciate your feedback:
          1. Moving that logic to CLIService would mean accessing some of the operation logic within CLIService, and from the general design of CLIService it looks like we wanted to avoid that.
          2. This will also mean that the async nature moves to Operation (runAsync, backgroundHandle) from SQLOperation.

          What are your thoughts? Thanks for the feedback and discussion so far!

          Show
          Vaibhav Gumashta added a comment - Carl Steinbach Uploaded a patch based on the feedback. Regarding moving the long polling timeout logic, I have some comments where I'll appreciate your feedback: 1. Moving that logic to CLIService would mean accessing some of the operation logic within CLIService, and from the general design of CLIService it looks like we wanted to avoid that. 2. This will also mean that the async nature moves to Operation (runAsync, backgroundHandle) from SQLOperation. What are your thoughts? Thanks for the feedback and discussion so far!
          Hide
          Carl Steinbach added a comment -

          I left some more comments on reviewboard.

          Are you planning to move the long polling timeout logic from SQLOperation.getState() to CLIService?

          Show
          Carl Steinbach added a comment - I left some more comments on reviewboard. Are you planning to move the long polling timeout logic from SQLOperation.getState() to CLIService?
          Hide
          Vaibhav Gumashta added a comment -

          Carl Steinbach Yes, you can take a look now. Thanks.

          Show
          Vaibhav Gumashta added a comment - Carl Steinbach Yes, you can take a look now. Thanks.
          Hide
          Carl Steinbach added a comment -

          [~vaibhavgumashta] Is the patch ready for review?

          Show
          Carl Steinbach added a comment - [~vaibhavgumashta] Is the patch ready for review?
          Hide
          Vaibhav Gumashta added a comment -

          Couldn't update the other rb link as I wasn't the creator. Link for latest changes: https://reviews.apache.org/r/15435/

          Show
          Vaibhav Gumashta added a comment - Couldn't update the other rb link as I wasn't the creator. Link for latest changes: https://reviews.apache.org/r/15435/
          Hide
          Vaibhav Gumashta added a comment -

          Thanks Carl Steinbach. I'll look the the feedback here and on rb.

          Meanwhile on your last comment: getOperationStatus returns after hive.server2.long.polling.timeout expires or the query completes - whichever happens first. So if polling.timeout = 5000 and the query finishes in 1000, the call to getOperationStatus will return in 1000. However, in the test case, the test query we are running is very short - therefore to test the behavior of the long polling timeout, we're explicitly setting the config to a smaller time than default, which is 5000; otherwise if the query takes 3000, it will return state FINISHED and we won't be able to test the timeout.

          Show
          Vaibhav Gumashta added a comment - Thanks Carl Steinbach . I'll look the the feedback here and on rb. Meanwhile on your last comment: getOperationStatus returns after hive.server2.long.polling.timeout expires or the query completes - whichever happens first. So if polling.timeout = 5000 and the query finishes in 1000, the call to getOperationStatus will return in 1000. However, in the test case, the test query we are running is very short - therefore to test the behavior of the long polling timeout, we're explicitly setting the config to a smaller time than default, which is 5000; otherwise if the query takes 3000, it will return state FINISHED and we won't be able to test the timeout.
          Hide
          Carl Steinbach added a comment -

          I posted the patch to reviewboard and left some comments: https://reviews.apache.org/r/15337/

          Based on the test case included with the patch it looks like hive.server2.long.polling.timeout sets a lower bound on the total round trip time required to execute a getOperationStatus RPC, e.g. if polling.timeout = 5000 my query can finish after a second but getOperationStatus will still block for another four seconds before returning. Is this accurate?

          Show
          Carl Steinbach added a comment - I posted the patch to reviewboard and left some comments: https://reviews.apache.org/r/15337/ Based on the test case included with the patch it looks like hive.server2.long.polling.timeout sets a lower bound on the total round trip time required to execute a getOperationStatus RPC, e.g. if polling.timeout = 5000 my query can finish after a second but getOperationStatus will still block for another four seconds before returning. Is this accurate?
          Hide
          Carl Steinbach added a comment -

          It wouldn't be too hard to add support for a non long polling getOperationStatus call in future, specially a non rpc call.

          Adding a getOperationStatusInternal() or getOperationStatusNonLongPoll() method to work around this problem is exactly the sort of thing I want to avoid. This patch implements a service layer feature, so I think it makes sense that the implementation belongs in the service layer as well. In CLIService.getOperationStatus() you have access to the SessionManager, and from that you can easily get the Session object, the Operation object, the operation type, and the Session's configuration. As an added bonus you also get the OperationHandle which makes it easier to emit useful log messages that reference the session and operation IDs.

          Show
          Carl Steinbach added a comment - It wouldn't be too hard to add support for a non long polling getOperationStatus call in future, specially a non rpc call. Adding a getOperationStatusInternal() or getOperationStatusNonLongPoll() method to work around this problem is exactly the sort of thing I want to avoid. This patch implements a service layer feature, so I think it makes sense that the implementation belongs in the service layer as well. In CLIService.getOperationStatus() you have access to the SessionManager, and from that you can easily get the Session object, the Operation object, the operation type, and the Session's configuration. As an added bonus you also get the OperationHandle which makes it easier to emit useful log messages that reference the session and operation IDs.
          Hide
          Thejas M Nair added a comment -

          Carl Steinbach What Vaibhav is saying sounds reasonable to me. It wouldn't be too hard to add support for a non long polling getOperationStatus call in future, specially a non rpc call. We can make the change when we have such a use case.

          Show
          Thejas M Nair added a comment - Carl Steinbach What Vaibhav is saying sounds reasonable to me. It wouldn't be too hard to add support for a non long polling getOperationStatus call in future, specially a non rpc call. We can make the change when we have such a use case.
          Hide
          Hive QA added a comment -

          Overall: +1 all checks pass

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12612471/HIVE-5217.D12801.6.patch

          SUCCESS: +1 4577 tests passed

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/168/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/168/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          

          This message is automatically generated.

          ATTACHMENT ID: 12612471

          Show
          Hive QA added a comment - Overall : +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12612471/HIVE-5217.D12801.6.patch SUCCESS: +1 4577 tests passed Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/168/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/168/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated. ATTACHMENT ID: 12612471
          Hide
          Vaibhav Gumashta added a comment -

          Patch rebased on trunk

          Show
          Vaibhav Gumashta added a comment - Patch rebased on trunk
          Hide
          Vaibhav Gumashta added a comment -

          Carl Steinbach Sure, let me rebase on the trunk. You can look at the patch here: https://reviews.facebook.net/D12801, but I can create a request on RB if you prefer that.

          Actually the delay period only affects the async SQLOperation (and returns immediately if the async call has the results) and doesn't change the blocking calls. Also, as far as I understand the use case you are pointing to is for a cleanup type of operation, where we might want to clean up the entire session based on whether the client is active or not. In that case, the cleanup thread might end up checking the last active time of the session rather than the operation state and based on whether it exceeds some threshold, delete the session from the server.

          Moving the long polling logic to CLIService might involve changing the Operation API, which I was trying to avoid until we get to a specific use case. But if you feel otherwise, I can definitely make the change.

          Thanks for looking at the patch!

          Show
          Vaibhav Gumashta added a comment - Carl Steinbach Sure, let me rebase on the trunk. You can look at the patch here: https://reviews.facebook.net/D12801 , but I can create a request on RB if you prefer that. Actually the delay period only affects the async SQLOperation (and returns immediately if the async call has the results) and doesn't change the blocking calls. Also, as far as I understand the use case you are pointing to is for a cleanup type of operation, where we might want to clean up the entire session based on whether the client is active or not. In that case, the cleanup thread might end up checking the last active time of the session rather than the operation state and based on whether it exceeds some threshold, delete the session from the server. Moving the long polling logic to CLIService might involve changing the Operation API, which I was trying to avoid until we get to a specific use case. But if you feel otherwise, I can definitely make the change. Thanks for looking at the patch!
          Hide
          Carl Steinbach added a comment -

          [~vaibhavgumashta] Version 5 of the patch won't apply cleanly on trunk. Can you please rebase the patch and post a review request on RB?

          One concern I have with the patch is that the delay period affects everyone who calls SQLOperation.getStatus(). Ideally we would be able to limit this behavior to remote clients only. I'm worried that at some point in the future we're going to want to have a housekeeping or monitoring thread that periodically calls getStatus() on all active operations, and we don't want getStatus() to block in these types of situations. Would it be practical to relocate this logic to the CLIService layer so that it only impacts CLIService clients?

          Show
          Carl Steinbach added a comment - [~vaibhavgumashta] Version 5 of the patch won't apply cleanly on trunk. Can you please rebase the patch and post a review request on RB? One concern I have with the patch is that the delay period affects everyone who calls SQLOperation.getStatus(). Ideally we would be able to limit this behavior to remote clients only. I'm worried that at some point in the future we're going to want to have a housekeeping or monitoring thread that periodically calls getStatus() on all active operations, and we don't want getStatus() to block in these types of situations. Would it be practical to relocate this logic to the CLIService layer so that it only impacts CLIService clients?
          Hide
          Vaibhav Gumashta added a comment -

          Sorry missed out attaching the patch. Thanks for the review!

          Show
          Vaibhav Gumashta added a comment - Sorry missed out attaching the patch. Thanks for the review!
          Hide
          Thejas M Nair added a comment -

          +1 for latest patch in phabricator. Can you please upload that patch to jira as well ?

          Show
          Thejas M Nair added a comment - +1 for latest patch in phabricator. Can you please upload that patch to jira as well ?
          Hide
          Vaibhav Gumashta added a comment -
          Show
          Vaibhav Gumashta added a comment - Phab link: https://reviews.facebook.net/D12801
          Show
          Vaibhav Gumashta added a comment - Here is the idea: http://tools.ietf.org/html/draft-loreto-http-bidirectional-07 .

            People

            • Assignee:
              Vaibhav Gumashta
              Reporter:
              Vaibhav Gumashta
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development