Hive
  1. Hive
  2. HIVE-6411

Support more generic way of using composite key for HBaseHandler

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.14.0
    • Component/s: HBase Handler
    • Labels:
    • Release Note:
      The new feature needs to be documented at Hive-HBase integration page.

      Description

      HIVE-2599 introduced using custom object for the row key. But it forces key objects to extend HBaseCompositeKey, which is again extension of LazyStruct. If user provides proper Object and OI, we can replace internal key and keyOI with those.

      Initial implementation is based on factory interface.

      public interface HBaseKeyFactory {
        void init(SerDeParameters parameters, Properties properties) throws SerDeException;
        ObjectInspector createObjectInspector(TypeInfo type) throws SerDeException;
        LazyObjectBase createObject(ObjectInspector inspector) throws SerDeException;
      }
      
      1. HIVE-6411.11.patch.txt
        189 kB
        Swarnim Kulkarni
      2. HIVE-6411.10.patch.txt
        189 kB
        Swarnim Kulkarni
      3. HIVE-6411.9.patch.txt
        188 kB
        Navis
      4. HIVE-6411.8.patch.txt
        178 kB
        Navis
      5. HIVE-6411.7.patch.txt
        172 kB
        Navis
      6. HIVE-6411.6.patch.txt
        100 kB
        Navis
      7. HIVE-6411.5.patch.txt
        93 kB
        Navis
      8. HIVE-6411.4.patch.txt
        94 kB
        Navis
      9. HIVE-6411.3.patch.txt
        85 kB
        Navis
      10. HIVE-6411.2.patch.txt
        53 kB
        Navis
      11. HIVE-6411.1.patch.txt
        46 kB
        Navis

        Issue Links

          Activity

          Hide
          Navis added a comment -

          Preliminary test

          Show
          Navis added a comment - Preliminary test
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12628439/HIVE-6411.1.patch.txt

          ERROR: -1 due to 2 failed/errored test(s), 5090 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_revoke_table_priv
          org.apache.hcatalog.hbase.snapshot.lock.TestWriteLock.testRun
          

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1303/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1303/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 2 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12628439

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12628439/HIVE-6411.1.patch.txt ERROR: -1 due to 2 failed/errored test(s), 5090 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_revoke_table_priv org.apache.hcatalog.hbase.snapshot.lock.TestWriteLock.testRun Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1303/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1303/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed This message is automatically generated. ATTACHMENT ID: 12628439
          Hide
          Navis added a comment -

          Cannot reproduce fail of TestWriteLock. authorization_revoke_table_priv is resolved in other issue. Brock Noland, Swarnim Kulkarni, Could you review this? (I cannot make review board entry by 502 error)

          Show
          Navis added a comment - Cannot reproduce fail of TestWriteLock. authorization_revoke_table_priv is resolved in other issue. Brock Noland , Swarnim Kulkarni , Could you review this? (I cannot make review board entry by 502 error)
          Hide
          Brock Noland added a comment -

          I also don't think the test failures are related. Xuefu Zhang also looked at the hbase composite key changes IIRC.

          Show
          Brock Noland added a comment - I also don't think the test failures are related. Xuefu Zhang also looked at the hbase composite key changes IIRC.
          Hide
          Swarnim Kulkarni added a comment -

          Navis Brock Noland I created a RB review here to make it easier to look at the diff. I'll start digging into the changes.

          [1] https://reviews.apache.org/r/18122/

          Show
          Swarnim Kulkarni added a comment - Navis Brock Noland I created a RB review here to make it easier to look at the diff. I'll start digging into the changes. [1] https://reviews.apache.org/r/18122/
          Hide
          Navis added a comment -

          Cannot update the patch, so made a new link. https://reviews.apache.org/r/18179/

          Show
          Navis added a comment - Cannot update the patch, so made a new link. https://reviews.apache.org/r/18179/
          Hide
          Hive QA added a comment -

          Overall: +1 all checks pass

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12631672/HIVE-6411.2.patch.txt

          SUCCESS: +1 5181 tests passed

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1559/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1559/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          

          This message is automatically generated.

          ATTACHMENT ID: 12631672

          Show
          Hive QA added a comment - Overall : +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12631672/HIVE-6411.2.patch.txt SUCCESS: +1 5181 tests passed Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1559/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1559/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated. ATTACHMENT ID: 12631672
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12632418/HIVE-6411.3.patch.txt

          ERROR: -1 due to 3 failed/errored test(s), 5240 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key2
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_auto_sortmerge_join_16
          org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testNegativeCliDriver_mapreduce_stack_trace_hadoop20
          

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1616/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1616/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 3 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12632418

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12632418/HIVE-6411.3.patch.txt ERROR: -1 due to 3 failed/errored test(s), 5240 tests executed Failed tests: org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key2 org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_auto_sortmerge_join_16 org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testNegativeCliDriver_mapreduce_stack_trace_hadoop20 Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1616/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1616/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 3 tests failed This message is automatically generated. ATTACHMENT ID: 12632418
          Hide
          Hive QA added a comment -

          Overall: -1 no tests executed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12632761/HIVE-6411.4.patch.txt

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1634/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1634/console

          Messages:

          **** This message was trimmed, see log for full details ****
          [INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ hive-cli ---
          [debug] execute contextualize
          [INFO] Using 'UTF-8' encoding to copy filtered resources.
          [INFO] skip non existing resourceDirectory /data/hive-ptest/working/apache-svn-trunk-source/cli/src/test/resources
          [INFO] 
          [INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ hive-cli ---
          [INFO] Executing tasks
          
          main:
              [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/cli/target/tmp
              [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/cli/target/warehouse
              [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/cli/target/tmp/conf
               [copy] Copying 5 files to /data/hive-ptest/working/apache-svn-trunk-source/cli/target/tmp/conf
          [INFO] Executed tasks
          [INFO] 
          [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ hive-cli ---
          [INFO] Compiling 4 source files to /data/hive-ptest/working/apache-svn-trunk-source/cli/target/test-classes
          [WARNING] Note: /data/hive-ptest/working/apache-svn-trunk-source/cli/src/test/org/apache/hadoop/hive/cli/TestCliDriverMethods.java uses unchecked or unsafe operations.
          [WARNING] Note: Recompile with -Xlint:unchecked for details.
          [INFO] 
          [INFO] --- maven-surefire-plugin:2.16:test (default-test) @ hive-cli ---
          [INFO] Tests are skipped.
          [INFO] 
          [INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ hive-cli ---
          [INFO] Building jar: /data/hive-ptest/working/apache-svn-trunk-source/cli/target/hive-cli-0.14.0-SNAPSHOT.jar
          [INFO] 
          [INFO] --- maven-install-plugin:2.4:install (default-install) @ hive-cli ---
          [INFO] Installing /data/hive-ptest/working/apache-svn-trunk-source/cli/target/hive-cli-0.14.0-SNAPSHOT.jar to /data/hive-ptest/working/maven/org/apache/hive/hive-cli/0.14.0-SNAPSHOT/hive-cli-0.14.0-SNAPSHOT.jar
          [INFO] Installing /data/hive-ptest/working/apache-svn-trunk-source/cli/pom.xml to /data/hive-ptest/working/maven/org/apache/hive/hive-cli/0.14.0-SNAPSHOT/hive-cli-0.14.0-SNAPSHOT.pom
          [INFO]                                                                         
          [INFO] ------------------------------------------------------------------------
          [INFO] Building Hive Contrib 0.14.0-SNAPSHOT
          [INFO] ------------------------------------------------------------------------
          [INFO] 
          [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-contrib ---
          [INFO] Deleting /data/hive-ptest/working/apache-svn-trunk-source/contrib (includes = [datanucleus.log, derby.log], excludes = [])
          [INFO] 
          [INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ hive-contrib ---
          [debug] execute contextualize
          [INFO] Using 'UTF-8' encoding to copy filtered resources.
          [INFO] skip non existing resourceDirectory /data/hive-ptest/working/apache-svn-trunk-source/contrib/src/main/resources
          [INFO] 
          [INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @ hive-contrib ---
          [INFO] Executing tasks
          
          main:
          [INFO] Executed tasks
          [INFO] 
          [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hive-contrib ---
          [INFO] Compiling 39 source files to /data/hive-ptest/working/apache-svn-trunk-source/contrib/target/classes
          [WARNING] Note: Some input files use or override a deprecated API.
          [WARNING] Note: Recompile with -Xlint:deprecation for details.
          [WARNING] Note: /data/hive-ptest/working/apache-svn-trunk-source/contrib/src/java/org/apache/hadoop/hive/contrib/udf/example/UDFExampleStructPrint.java uses unchecked or unsafe operations.
          [WARNING] Note: Recompile with -Xlint:unchecked for details.
          [INFO] 
          [INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ hive-contrib ---
          [debug] execute contextualize
          [INFO] Using 'UTF-8' encoding to copy filtered resources.
          [INFO] skip non existing resourceDirectory /data/hive-ptest/working/apache-svn-trunk-source/contrib/src/test/resources
          [INFO] 
          [INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ hive-contrib ---
          [INFO] Executing tasks
          
          main:
              [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/contrib/target/tmp
              [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/contrib/target/warehouse
              [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/contrib/target/tmp/conf
               [copy] Copying 5 files to /data/hive-ptest/working/apache-svn-trunk-source/contrib/target/tmp/conf
          [INFO] Executed tasks
          [INFO] 
          [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ hive-contrib ---
          [INFO] Compiling 2 source files to /data/hive-ptest/working/apache-svn-trunk-source/contrib/target/test-classes
          [WARNING] Note: /data/hive-ptest/working/apache-svn-trunk-source/contrib/src/test/org/apache/hadoop/hive/contrib/serde2/TestRegexSerDe.java uses or overrides a deprecated API.
          [WARNING] Note: Recompile with -Xlint:deprecation for details.
          [INFO] 
          [INFO] --- maven-surefire-plugin:2.16:test (default-test) @ hive-contrib ---
          [INFO] Tests are skipped.
          [INFO] 
          [INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ hive-contrib ---
          [INFO] Building jar: /data/hive-ptest/working/apache-svn-trunk-source/contrib/target/hive-contrib-0.14.0-SNAPSHOT.jar
          [INFO] 
          [INFO] --- maven-install-plugin:2.4:install (default-install) @ hive-contrib ---
          [INFO] Installing /data/hive-ptest/working/apache-svn-trunk-source/contrib/target/hive-contrib-0.14.0-SNAPSHOT.jar to /data/hive-ptest/working/maven/org/apache/hive/hive-contrib/0.14.0-SNAPSHOT/hive-contrib-0.14.0-SNAPSHOT.jar
          [INFO] Installing /data/hive-ptest/working/apache-svn-trunk-source/contrib/pom.xml to /data/hive-ptest/working/maven/org/apache/hive/hive-contrib/0.14.0-SNAPSHOT/hive-contrib-0.14.0-SNAPSHOT.pom
          [INFO]                                                                         
          [INFO] ------------------------------------------------------------------------
          [INFO] Building Hive HBase Handler 0.14.0-SNAPSHOT
          [INFO] ------------------------------------------------------------------------
          [INFO] 
          [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-hbase-handler ---
          [INFO] Deleting /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler (includes = [datanucleus.log, derby.log], excludes = [])
          [INFO] 
          [INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ hive-hbase-handler ---
          [debug] execute contextualize
          [INFO] Using 'UTF-8' encoding to copy filtered resources.
          [INFO] skip non existing resourceDirectory /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/main/resources
          [INFO] 
          [INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @ hive-hbase-handler ---
          [INFO] Executing tasks
          
          main:
          [INFO] Executed tasks
          [INFO] 
          [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hive-hbase-handler ---
          [INFO] Compiling 22 source files to /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/target/classes
          [WARNING] Note: Some input files use or override a deprecated API.
          [WARNING] Note: Recompile with -Xlint:deprecation for details.
          [WARNING] Note: Some input files use unchecked or unsafe operations.
          [WARNING] Note: Recompile with -Xlint:unchecked for details.
          [INFO] 
          [INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ hive-hbase-handler ---
          [debug] execute contextualize
          [INFO] Using 'UTF-8' encoding to copy filtered resources.
          [INFO] skip non existing resourceDirectory /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/test/resources
          [INFO] 
          [INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ hive-hbase-handler ---
          [INFO] Executing tasks
          
          main:
              [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/target/tmp
              [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/target/warehouse
              [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/target/tmp/conf
               [copy] Copying 5 files to /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/target/tmp/conf
          [INFO] Executed tasks
          [INFO] 
          [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ hive-hbase-handler ---
          [INFO] Compiling 6 source files to /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/target/test-classes
          [INFO] -------------------------------------------------------------
          [WARNING] COMPILATION WARNING : 
          [INFO] -------------------------------------------------------------
          [WARNING] Note: Some input files use or override a deprecated API.
          [WARNING] Note: Recompile with -Xlint:deprecation for details.
          [INFO] 2 warnings 
          [INFO] -------------------------------------------------------------
          [INFO] -------------------------------------------------------------
          [ERROR] COMPILATION ERROR : 
          [INFO] -------------------------------------------------------------
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseKeyFactory.java:[79,10] cannot find symbol
          symbol  : class DecomposedPredicate
          location: class org.apache.hadoop.hive.hbase.TestHBaseKeyFactory
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseKeyFactory.java:[78,3] method does not override or implement a method from a supertype
          [INFO] 2 errors 
          [INFO] -------------------------------------------------------------
          [INFO] ------------------------------------------------------------------------
          [INFO] Reactor Summary:
          [INFO] 
          [INFO] Hive .............................................. SUCCESS [3.395s]
          [INFO] Hive Ant Utilities ................................ SUCCESS [7.360s]
          [INFO] Hive Shims Common ................................. SUCCESS [3.380s]
          [INFO] Hive Shims 0.20 ................................... SUCCESS [2.307s]
          [INFO] Hive Shims Secure Common .......................... SUCCESS [2.864s]
          [INFO] Hive Shims 0.20S .................................. SUCCESS [1.834s]
          [INFO] Hive Shims 0.23 ................................... SUCCESS [4.198s]
          [INFO] Hive Shims ........................................ SUCCESS [0.601s]
          [INFO] Hive Common ....................................... SUCCESS [10.229s]
          [INFO] Hive Serde ........................................ SUCCESS [9.177s]
          [INFO] Hive Metastore .................................... SUCCESS [32.524s]
          [INFO] Hive Query Language ............................... SUCCESS [1:16.323s]
          [INFO] Hive Service ...................................... SUCCESS [6.563s]
          [INFO] Hive JDBC ......................................... SUCCESS [2.362s]
          [INFO] Hive Beeline ...................................... SUCCESS [1.536s]
          [INFO] Hive CLI .......................................... SUCCESS [2.068s]
          [INFO] Hive Contrib ...................................... SUCCESS [1.177s]
          [INFO] Hive HBase Handler ................................ FAILURE [2.615s]
          [INFO] Hive HCatalog ..................................... SKIPPED
          [INFO] Hive HCatalog Core ................................ SKIPPED
          [INFO] Hive HCatalog Pig Adapter ......................... SKIPPED
          [INFO] Hive HCatalog Server Extensions ................... SKIPPED
          [INFO] Hive HCatalog Webhcat Java Client ................. SKIPPED
          [INFO] Hive HCatalog Webhcat ............................. SKIPPED
          [INFO] Hive HCatalog HBase Storage Handler ............... SKIPPED
          [INFO] Hive HWI .......................................... SKIPPED
          [INFO] Hive ODBC ......................................... SKIPPED
          [INFO] Hive Shims Aggregator ............................. SKIPPED
          [INFO] Hive TestUtils .................................... SKIPPED
          [INFO] Hive Packaging .................................... SKIPPED
          [INFO] ------------------------------------------------------------------------
          [INFO] BUILD FAILURE
          [INFO] ------------------------------------------------------------------------
          [INFO] Total time: 2:55.077s
          [INFO] Finished at: Wed Mar 05 18:00:49 EST 2014
          [INFO] Final Memory: 75M/490M
          [INFO] ------------------------------------------------------------------------
          [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile (default-testCompile) on project hive-hbase-handler: Compilation failure: Compilation failure:
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseKeyFactory.java:[79,10] cannot find symbol
          [ERROR] symbol  : class DecomposedPredicate
          [ERROR] location: class org.apache.hadoop.hive.hbase.TestHBaseKeyFactory
          [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseKeyFactory.java:[78,3] method does not override or implement a method from a supertype
          [ERROR] -> [Help 1]
          [ERROR] 
          [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
          [ERROR] Re-run Maven using the -X switch to enable full debug logging.
          [ERROR] 
          [ERROR] For more information about the errors and possible solutions, please read the following articles:
          [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
          [ERROR] 
          [ERROR] After correcting the problems, you can resume the build with the command
          [ERROR]   mvn <goals> -rf :hive-hbase-handler
          + exit 1
          '
          

          This message is automatically generated.

          ATTACHMENT ID: 12632761

          Show
          Hive QA added a comment - Overall : -1 no tests executed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12632761/HIVE-6411.4.patch.txt Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1634/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1634/console Messages: **** This message was trimmed, see log for full details **** [INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ hive-cli --- [debug] execute contextualize [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /data/hive-ptest/working/apache-svn-trunk-source/cli/src/test/resources [INFO] [INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ hive-cli --- [INFO] Executing tasks main: [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/cli/target/tmp [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/cli/target/warehouse [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/cli/target/tmp/conf [copy] Copying 5 files to /data/hive-ptest/working/apache-svn-trunk-source/cli/target/tmp/conf [INFO] Executed tasks [INFO] [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ hive-cli --- [INFO] Compiling 4 source files to /data/hive-ptest/working/apache-svn-trunk-source/cli/target/test-classes [WARNING] Note: /data/hive-ptest/working/apache-svn-trunk-source/cli/src/test/org/apache/hadoop/hive/cli/TestCliDriverMethods.java uses unchecked or unsafe operations. [WARNING] Note: Recompile with -Xlint:unchecked for details. [INFO] [INFO] --- maven-surefire-plugin:2.16:test (default-test) @ hive-cli --- [INFO] Tests are skipped. [INFO] [INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ hive-cli --- [INFO] Building jar: /data/hive-ptest/working/apache-svn-trunk-source/cli/target/hive-cli-0.14.0-SNAPSHOT.jar [INFO] [INFO] --- maven-install-plugin:2.4:install (default-install) @ hive-cli --- [INFO] Installing /data/hive-ptest/working/apache-svn-trunk-source/cli/target/hive-cli-0.14.0-SNAPSHOT.jar to /data/hive-ptest/working/maven/org/apache/hive/hive-cli/0.14.0-SNAPSHOT/hive-cli-0.14.0-SNAPSHOT.jar [INFO] Installing /data/hive-ptest/working/apache-svn-trunk-source/cli/pom.xml to /data/hive-ptest/working/maven/org/apache/hive/hive-cli/0.14.0-SNAPSHOT/hive-cli-0.14.0-SNAPSHOT.pom [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building Hive Contrib 0.14.0-SNAPSHOT [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-contrib --- [INFO] Deleting /data/hive-ptest/working/apache-svn-trunk-source/contrib (includes = [datanucleus.log, derby.log], excludes = []) [INFO] [INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ hive-contrib --- [debug] execute contextualize [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /data/hive-ptest/working/apache-svn-trunk-source/contrib/src/main/resources [INFO] [INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @ hive-contrib --- [INFO] Executing tasks main: [INFO] Executed tasks [INFO] [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hive-contrib --- [INFO] Compiling 39 source files to /data/hive-ptest/working/apache-svn-trunk-source/contrib/target/classes [WARNING] Note: Some input files use or override a deprecated API. [WARNING] Note: Recompile with -Xlint:deprecation for details. [WARNING] Note: /data/hive-ptest/working/apache-svn-trunk-source/contrib/src/java/org/apache/hadoop/hive/contrib/udf/example/UDFExampleStructPrint.java uses unchecked or unsafe operations. [WARNING] Note: Recompile with -Xlint:unchecked for details. [INFO] [INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ hive-contrib --- [debug] execute contextualize [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /data/hive-ptest/working/apache-svn-trunk-source/contrib/src/test/resources [INFO] [INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ hive-contrib --- [INFO] Executing tasks main: [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/contrib/target/tmp [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/contrib/target/warehouse [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/contrib/target/tmp/conf [copy] Copying 5 files to /data/hive-ptest/working/apache-svn-trunk-source/contrib/target/tmp/conf [INFO] Executed tasks [INFO] [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ hive-contrib --- [INFO] Compiling 2 source files to /data/hive-ptest/working/apache-svn-trunk-source/contrib/target/test-classes [WARNING] Note: /data/hive-ptest/working/apache-svn-trunk-source/contrib/src/test/org/apache/hadoop/hive/contrib/serde2/TestRegexSerDe.java uses or overrides a deprecated API. [WARNING] Note: Recompile with -Xlint:deprecation for details. [INFO] [INFO] --- maven-surefire-plugin:2.16:test (default-test) @ hive-contrib --- [INFO] Tests are skipped. [INFO] [INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ hive-contrib --- [INFO] Building jar: /data/hive-ptest/working/apache-svn-trunk-source/contrib/target/hive-contrib-0.14.0-SNAPSHOT.jar [INFO] [INFO] --- maven-install-plugin:2.4:install (default-install) @ hive-contrib --- [INFO] Installing /data/hive-ptest/working/apache-svn-trunk-source/contrib/target/hive-contrib-0.14.0-SNAPSHOT.jar to /data/hive-ptest/working/maven/org/apache/hive/hive-contrib/0.14.0-SNAPSHOT/hive-contrib-0.14.0-SNAPSHOT.jar [INFO] Installing /data/hive-ptest/working/apache-svn-trunk-source/contrib/pom.xml to /data/hive-ptest/working/maven/org/apache/hive/hive-contrib/0.14.0-SNAPSHOT/hive-contrib-0.14.0-SNAPSHOT.pom [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building Hive HBase Handler 0.14.0-SNAPSHOT [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-hbase-handler --- [INFO] Deleting /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler (includes = [datanucleus.log, derby.log], excludes = []) [INFO] [INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ hive-hbase-handler --- [debug] execute contextualize [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/main/resources [INFO] [INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @ hive-hbase-handler --- [INFO] Executing tasks main: [INFO] Executed tasks [INFO] [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hive-hbase-handler --- [INFO] Compiling 22 source files to /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/target/classes [WARNING] Note: Some input files use or override a deprecated API. [WARNING] Note: Recompile with -Xlint:deprecation for details. [WARNING] Note: Some input files use unchecked or unsafe operations. [WARNING] Note: Recompile with -Xlint:unchecked for details. [INFO] [INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ hive-hbase-handler --- [debug] execute contextualize [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/test/resources [INFO] [INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ hive-hbase-handler --- [INFO] Executing tasks main: [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/target/tmp [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/target/warehouse [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/target/tmp/conf [copy] Copying 5 files to /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/target/tmp/conf [INFO] Executed tasks [INFO] [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ hive-hbase-handler --- [INFO] Compiling 6 source files to /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/target/test-classes [INFO] ------------------------------------------------------------- [WARNING] COMPILATION WARNING : [INFO] ------------------------------------------------------------- [WARNING] Note: Some input files use or override a deprecated API. [WARNING] Note: Recompile with -Xlint:deprecation for details. [INFO] 2 warnings [INFO] ------------------------------------------------------------- [INFO] ------------------------------------------------------------- [ERROR] COMPILATION ERROR : [INFO] ------------------------------------------------------------- [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseKeyFactory.java:[79,10] cannot find symbol symbol : class DecomposedPredicate location: class org.apache.hadoop.hive.hbase.TestHBaseKeyFactory [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseKeyFactory.java:[78,3] method does not override or implement a method from a supertype [INFO] 2 errors [INFO] ------------------------------------------------------------- [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Hive .............................................. SUCCESS [3.395s] [INFO] Hive Ant Utilities ................................ SUCCESS [7.360s] [INFO] Hive Shims Common ................................. SUCCESS [3.380s] [INFO] Hive Shims 0.20 ................................... SUCCESS [2.307s] [INFO] Hive Shims Secure Common .......................... SUCCESS [2.864s] [INFO] Hive Shims 0.20S .................................. SUCCESS [1.834s] [INFO] Hive Shims 0.23 ................................... SUCCESS [4.198s] [INFO] Hive Shims ........................................ SUCCESS [0.601s] [INFO] Hive Common ....................................... SUCCESS [10.229s] [INFO] Hive Serde ........................................ SUCCESS [9.177s] [INFO] Hive Metastore .................................... SUCCESS [32.524s] [INFO] Hive Query Language ............................... SUCCESS [1:16.323s] [INFO] Hive Service ...................................... SUCCESS [6.563s] [INFO] Hive JDBC ......................................... SUCCESS [2.362s] [INFO] Hive Beeline ...................................... SUCCESS [1.536s] [INFO] Hive CLI .......................................... SUCCESS [2.068s] [INFO] Hive Contrib ...................................... SUCCESS [1.177s] [INFO] Hive HBase Handler ................................ FAILURE [2.615s] [INFO] Hive HCatalog ..................................... SKIPPED [INFO] Hive HCatalog Core ................................ SKIPPED [INFO] Hive HCatalog Pig Adapter ......................... SKIPPED [INFO] Hive HCatalog Server Extensions ................... SKIPPED [INFO] Hive HCatalog Webhcat Java Client ................. SKIPPED [INFO] Hive HCatalog Webhcat ............................. SKIPPED [INFO] Hive HCatalog HBase Storage Handler ............... SKIPPED [INFO] Hive HWI .......................................... SKIPPED [INFO] Hive ODBC ......................................... SKIPPED [INFO] Hive Shims Aggregator ............................. SKIPPED [INFO] Hive TestUtils .................................... SKIPPED [INFO] Hive Packaging .................................... SKIPPED [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 2:55.077s [INFO] Finished at: Wed Mar 05 18:00:49 EST 2014 [INFO] Final Memory: 75M/490M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile (default-testCompile) on project hive-hbase-handler: Compilation failure: Compilation failure: [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseKeyFactory.java:[79,10] cannot find symbol [ERROR] symbol : class DecomposedPredicate [ERROR] location: class org.apache.hadoop.hive.hbase.TestHBaseKeyFactory [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseKeyFactory.java:[78,3] method does not override or implement a method from a supertype [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn <goals> -rf :hive-hbase-handler + exit 1 ' This message is automatically generated. ATTACHMENT ID: 12632761
          Hide
          Xuefu Zhang added a comment -

          Navis, would you mind update the review board with the latest patch? Thanks.

          Show
          Xuefu Zhang added a comment - Navis , would you mind update the review board with the latest patch? Thanks.
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12632985/HIVE-6411.5.patch.txt

          ERROR: -1 due to 6 failed/errored test(s), 5360 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key2
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_auto_sortmerge_join_16
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_quotedid_smb
          org.apache.hive.beeline.TestSchemaTool.testSchemaInit
          org.apache.hive.beeline.TestSchemaTool.testSchemaUpgrade
          org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testExecuteStatementAsync
          

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1642/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1642/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 6 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12632985

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12632985/HIVE-6411.5.patch.txt ERROR: -1 due to 6 failed/errored test(s), 5360 tests executed Failed tests: org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key2 org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_auto_sortmerge_join_16 org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_quotedid_smb org.apache.hive.beeline.TestSchemaTool.testSchemaInit org.apache.hive.beeline.TestSchemaTool.testSchemaUpgrade org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testExecuteStatementAsync Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1642/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1642/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 6 tests failed This message is automatically generated. ATTACHMENT ID: 12632985
          Hide
          Navis added a comment -

          Removed queries resulting binary output in out file

          Show
          Navis added a comment - Removed queries resulting binary output in out file
          Hide
          Xuefu Zhang added a comment -

          Thanks for the nice work. I haven't finished review yet, but posted some comments on RB.

          Show
          Xuefu Zhang added a comment - Thanks for the nice work. I haven't finished review yet, but posted some comments on RB.
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12633301/HIVE-6411.6.patch.txt

          ERROR: -1 due to 1 failed/errored test(s), 5375 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_auto_sortmerge_join_16
          

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1662/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1662/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 1 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12633301

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12633301/HIVE-6411.6.patch.txt ERROR: -1 due to 1 failed/errored test(s), 5375 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_auto_sortmerge_join_16 Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1662/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1662/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed This message is automatically generated. ATTACHMENT ID: 12633301
          Hide
          Xuefu Zhang added a comment -

          I left more comments on RB.

          BTW, I noticed that RB has a new patch, which isn't attached here to run test.

          Show
          Xuefu Zhang added a comment - I left more comments on RB. BTW, I noticed that RB has a new patch, which isn't attached here to run test.
          Hide
          Xuefu Zhang added a comment -

          Navis Swarnim Kulkarni I'm wondering if we could move this forward?

          Show
          Xuefu Zhang added a comment - Navis Swarnim Kulkarni I'm wondering if we could move this forward?
          Hide
          Swarnim Kulkarni added a comment -

          I think there are some RB comments that need to be addressed on this with some refactoring. Navis I know you are already working on this patch. Please so let me know if you need some assistance from my side on this. Thanks.

          Show
          Swarnim Kulkarni added a comment - I think there are some RB comments that need to be addressed on this with some refactoring. Navis I know you are already working on this patch. Please so let me know if you need some assistance from my side on this. Thanks.
          Hide
          Xuefu Zhang added a comment -

          Navis Given that HIVE-6677 is in trunk, I'm wondering if you'd like to move this forward. I understand that the patch will need rebasing and somewhat refactoring, but I think either Swarnim Kulkarni or I can put some help on that if needed. Thanks for your effort on this.

          Show
          Xuefu Zhang added a comment - Navis Given that HIVE-6677 is in trunk, I'm wondering if you'd like to move this forward. I understand that the patch will need rebasing and somewhat refactoring, but I think either Swarnim Kulkarni or I can put some help on that if needed. Thanks for your effort on this.
          Hide
          Swarnim Kulkarni added a comment -

          Would it make sense at all to make sense to log another bug for the refactoring changes and moving forward with this patch minus the refactoring that needs to be done?

          Show
          Swarnim Kulkarni added a comment - Would it make sense at all to make sense to log another bug for the refactoring changes and moving forward with this patch minus the refactoring that needs to be done?
          Hide
          Xuefu Zhang added a comment -

          Because of HIVE-6677, this patch needs some rework anyway, I'd think we it might be better we just get it done in one chunk. It seems Navis isn't available for this, but if anyone would like to take this forward, please communicate and feel free to do so.

          Show
          Xuefu Zhang added a comment - Because of HIVE-6677 , this patch needs some rework anyway, I'd think we it might be better we just get it done in one chunk. It seems Navis isn't available for this, but if anyone would like to take this forward, please communicate and feel free to do so.
          Hide
          Swarnim Kulkarni added a comment -

          Navis If you are fine with it, I can pick this up from where you left it off. Let me know.

          Show
          Swarnim Kulkarni added a comment - Navis If you are fine with it, I can pick this up from where you left it off. Let me know.
          Hide
          Navis added a comment -

          Swarnim Kulkarni Xuefu Zhang I've done rebasing a few days ago, with some refactorings. I think I've done a little much of that. Anyway, will update review board shortly after.

          Show
          Navis added a comment - Swarnim Kulkarni Xuefu Zhang I've done rebasing a few days ago, with some refactorings. I think I've done a little much of that. Anyway, will update review board shortly after.
          Hide
          Xuefu Zhang added a comment -

          Thanks for the great work, Navis. The patch looks good. I left some minor comments on the RB.

          One thing that I don't quite get is the changes related to predicate pushdown. Is this necessary? It would be great if someone can take a look at that. Otherwise, I'll dig more.

          Show
          Xuefu Zhang added a comment - Thanks for the great work, Navis . The patch looks good. I left some minor comments on the RB. One thing that I don't quite get is the changes related to predicate pushdown. Is this necessary? It would be great if someone can take a look at that. Otherwise, I'll dig more.
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12637969/HIVE-6411.8.patch.txt

          ERROR: -1 due to 1 failed/errored test(s), 5515 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_auto_sortmerge_join_16
          

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2063/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2063/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 1 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12637969

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12637969/HIVE-6411.8.patch.txt ERROR: -1 due to 1 failed/errored test(s), 5515 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_auto_sortmerge_join_16 Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2063/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2063/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed This message is automatically generated. ATTACHMENT ID: 12637969
          Hide
          Swarnim Kulkarni added a comment -

          +1 from me.

          Show
          Swarnim Kulkarni added a comment - +1 from me.
          Hide
          Hive QA added a comment -

          Overall: +1 all checks pass

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12638202/HIVE-6411.9.patch.txt

          SUCCESS: +1 5541 tests passed

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2081/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2081/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          

          This message is automatically generated.

          ATTACHMENT ID: 12638202

          Show
          Hive QA added a comment - Overall : +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12638202/HIVE-6411.9.patch.txt SUCCESS: +1 5541 tests passed Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2081/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/2081/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated. ATTACHMENT ID: 12638202
          Hide
          Swarnim Kulkarni added a comment -

          Xuefu ZhangNavis I was wondering if there is something that is still holding us back on this JIRA or can we get this in?

          Show
          Swarnim Kulkarni added a comment - Xuefu Zhang Navis I was wondering if there is something that is still holding us back on this JIRA or can we get this in?
          Hide
          Xuefu Zhang added a comment -

          The only thing that's holding this is time: I haven't got a chance to go over the latest patch yet. One thing that I'm unclear about is the predicate pushdown related changes, per my previous comments.

          One thing that I don't quite get is the changes related to predicate pushdown. Is this necessary?

          An explanation of the nature of the change should help. Otherwise, it may take more of my time digging around.

          Show
          Xuefu Zhang added a comment - The only thing that's holding this is time: I haven't got a chance to go over the latest patch yet. One thing that I'm unclear about is the predicate pushdown related changes, per my previous comments. One thing that I don't quite get is the changes related to predicate pushdown. Is this necessary? An explanation of the nature of the change should help. Otherwise, it may take more of my time digging around.
          Hide
          Swarnim Kulkarni added a comment -

          I think part of that came from merging of my patch from https://issues.apache.org/jira/browse/HIVE-6290 which added a support for filter pushdown for structs.

          Show
          Swarnim Kulkarni added a comment - I think part of that came from merging of my patch from https://issues.apache.org/jira/browse/HIVE-6290 which added a support for filter pushdown for structs.
          Hide
          Xuefu Zhang added a comment -

          Actually I was referring to the changes made on the second page of review diff, such as serialization a filter object. I also saw some seemingly dead code. I guess I just need to spend a little bit more time understanding this.

          Show
          Xuefu Zhang added a comment - Actually I was referring to the changes made on the second page of review diff, such as serialization a filter object. I also saw some seemingly dead code. I guess I just need to spend a little bit more time understanding this.
          Hide
          Swarnim Kulkarni added a comment -

          Xuefu Zhang, Navis Should we move ahead with this or is there something blocking us?

          Show
          Swarnim Kulkarni added a comment - Xuefu Zhang , Navis Should we move ahead with this or is there something blocking us?
          Hide
          Xuefu Zhang added a comment -

          I believe there are some minor items to be resolved on review board.

          Show
          Xuefu Zhang added a comment - I believe there are some minor items to be resolved on review board.
          Hide
          Swarnim Kulkarni added a comment -

          Navis Please do let me know if there is anything that I can help out with. I think we should move ahead with this soon

          Show
          Swarnim Kulkarni added a comment - Navis Please do let me know if there is anything that I can help out with. I think we should move ahead with this soon
          Hide
          Xuefu Zhang added a comment -

          Swarnim Kulkarni If you look at the latest review comment, you will see one main item on which your help might be helpful. It's about the changes made when processing exprnode. I don't quite follow the change, so some documentation might help.

          Show
          Xuefu Zhang added a comment - Swarnim Kulkarni If you look at the latest review comment, you will see one main item on which your help might be helpful. It's about the changes made when processing exprnode. I don't quite follow the change, so some documentation might help.
          Hide
          Swarnim Kulkarni added a comment -

          I added comments on the RB.

          Show
          Swarnim Kulkarni added a comment - I added comments on the RB.
          Hide
          Swarnim Kulkarni added a comment -

          Is it possible for us to get this in by the end of the week or so? The only reason I ask is I have a bunch of other patches as well that I wanted to submit but are currently blocked by this JIRA. . Thanks!

          Show
          Swarnim Kulkarni added a comment - Is it possible for us to get this in by the end of the week or so? The only reason I ask is I have a bunch of other patches as well that I wanted to submit but are currently blocked by this JIRA. . Thanks!
          Hide
          Xuefu Zhang added a comment -

          Swarnim Kulkarni It seems Navis might not be available for this at the moment. If you wish, you could help update the patch. Please communicate if you choose to do so.

          Show
          Xuefu Zhang added a comment - Swarnim Kulkarni It seems Navis might not be available for this at the moment. If you wish, you could help update the patch. Please communicate if you choose to do so.
          Hide
          Swarnim Kulkarni added a comment -

          Sounds good to me.

          Navis Please do let me know if you will get a chance to look at it or else I can pick it up over the weekend and start addressing remaining comments on the RB.

          Thanks!

          Show
          Swarnim Kulkarni added a comment - Sounds good to me. Navis Please do let me know if you will get a chance to look at it or else I can pick it up over the weekend and start addressing remaining comments on the RB. Thanks!
          Hide
          Swarnim Kulkarni added a comment -

          Updated patch to address Xuefu Zhang's review comments and also rebased with the current master.

          Show
          Swarnim Kulkarni added a comment - Updated patch to address Xuefu Zhang 's review comments and also rebased with the current master.
          Hide
          Swarnim Kulkarni added a comment -

          Unfortunately since I couldn't update the old RB, I created a new one[1]. Xuefu Zhang I think I addressed all concerns on the patch that you had with this new one. Please let me know if I missed something.

          [1] https://reviews.apache.org/r/21138/

          Show
          Swarnim Kulkarni added a comment - Unfortunately since I couldn't update the old RB, I created a new one [1] . Xuefu Zhang I think I addressed all concerns on the patch that you had with this new one. Please let me know if I missed something. [1] https://reviews.apache.org/r/21138/
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12643656/HIVE-6411.10.patch.txt

          ERROR: -1 due to 8 failed/errored test(s), 5497 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_partscan_1_23
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_bucket_map_join_tez1
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_optimization
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_insert1
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_load_dyn_part1
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_union
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_stats_counter_partitioned
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/134/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/134/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 8 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12643656

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12643656/HIVE-6411.10.patch.txt ERROR: -1 due to 8 failed/errored test(s), 5497 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_partscan_1_23 org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_bucket_map_join_tez1 org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_optimization org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_insert1 org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_load_dyn_part1 org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_union org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_stats_counter_partitioned Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/134/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/134/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 8 tests failed This message is automatically generated. ATTACHMENT ID: 12643656
          Hide
          Swarnim Kulkarni added a comment -

          The above test failures don't seem related to any of my changes with the new patch.

          Show
          Swarnim Kulkarni added a comment - The above test failures don't seem related to any of my changes with the new patch.
          Hide
          Xuefu Zhang added a comment -

          I left a few comments on review board, mostly about documentation or code comments that help me understand the changes.

          Show
          Xuefu Zhang added a comment - I left a few comments on review board, mostly about documentation or code comments that help me understand the changes.
          Hide
          Swarnim Kulkarni added a comment -

          Comments addressed.

          Show
          Swarnim Kulkarni added a comment - Comments addressed.
          Hide
          Swarnim Kulkarni added a comment -

          RB updated.

          Show
          Swarnim Kulkarni added a comment - RB updated.
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12643959/HIVE-6411.11.patch.txt

          ERROR: -1 due to 2 failed/errored test(s), 5438 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_partscan_1_23
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/166/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/166/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 2 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12643959

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12643959/HIVE-6411.11.patch.txt ERROR: -1 due to 2 failed/errored test(s), 5438 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_partscan_1_23 org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/166/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/166/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed This message is automatically generated. ATTACHMENT ID: 12643959
          Hide
          Swarnim Kulkarni added a comment -

          Xuefu Zhang I have responded to your comments on the RB.

          Navis If you get a chance, would you mind telling us about your choice of a FamilyFilter here[1]. Use of family filter for row key scans doesn't seem appropriate to us.

          [1] https://reviews.apache.org/r/21138/diff/1/?file=575776#file575776line132

          Show
          Swarnim Kulkarni added a comment - Xuefu Zhang I have responded to your comments on the RB. Navis If you get a chance, would you mind telling us about your choice of a FamilyFilter here [1] . Use of family filter for row key scans doesn't seem appropriate to us. [1] https://reviews.apache.org/r/21138/diff/1/?file=575776#file575776line132
          Hide
          Xuefu Zhang added a comment -

          +1

          @Swarnim K Please log a followup JIRA to track the FamilyFilter issue and link it here.

          Show
          Xuefu Zhang added a comment - +1 @Swarnim K Please log a followup JIRA to track the FamilyFilter issue and link it here.
          Hide
          Swarnim Kulkarni added a comment -

          Xuefu Zhang Done. I have also linked to this newly created issue.

          [1] https://issues.apache.org/jira/browse/HIVE-7048

          Show
          Swarnim Kulkarni added a comment - Xuefu Zhang Done. I have also linked to this newly created issue. [1] https://issues.apache.org/jira/browse/HIVE-7048
          Hide
          Xuefu Zhang added a comment -

          Patch committed to trunk. Thanks to Navis and Swarnim for working on the patch.

          Show
          Xuefu Zhang added a comment - Patch committed to trunk. Thanks to Navis and Swarnim for working on the patch.
          Hide
          Swarnim Kulkarni added a comment -

          Thanks for the review Xuefu. I think we can now mark HIVE-6290 as resolved as well as the patch for that was included as a part of this JIRA.

          Show
          Swarnim Kulkarni added a comment - Thanks for the review Xuefu. I think we can now mark HIVE-6290 as resolved as well as the patch for that was included as a part of this JIRA.
          Hide
          Lefty Leverenz added a comment -

          The release note says this should be documented at the Hive-HBase Integration page, which is in the Design Docs:

          Show
          Lefty Leverenz added a comment - The release note says this should be documented at the Hive-HBase Integration page, which is in the Design Docs: Design Docs – Completed: Hive HBase Integration
          Hide
          Gautam Kowshik added a comment -

          Have we tried to back port this to Hive .13 .. This is a very useful feature to have for hive over hbase in the current stable version as well. If not i can try looking into this in a separate jira.

          Show
          Gautam Kowshik added a comment - Have we tried to back port this to Hive .13 .. This is a very useful feature to have for hive over hbase in the current stable version as well. If not i can try looking into this in a separate jira.
          Hide
          Thejas M Nair added a comment -

          This has been fixed in 0.14 release. Please open new jira if you see any issues.

          Show
          Thejas M Nair added a comment - This has been fixed in 0.14 release. Please open new jira if you see any issues.

            People

            • Assignee:
              Navis
              Reporter:
              Navis
            • Votes:
              1 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development