Hive
  1. Hive
  2. HIVE-5567

Add better protection code for SARGs

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.12.0
    • Fix Version/s: 0.13.0
    • Component/s: File Formats
    • Labels:
      None

      Description

      Currently, the SARG parser gets a NPE when the push down predicate uses a type like decimal that isn't supported.

      1. HIVE-5567.patch
        62 kB
        Owen O'Malley
      2. HIVE-5567.patch
        62 kB
        Owen O'Malley
      3. HIVE-5567.patch
        62 kB
        Owen O'Malley
      4. HIVE-5567.patch
        376 kB
        Owen O'Malley
      5. D13479.3.patch
        37 kB
        Phabricator
      6. D13479.2.patch
        37 kB
        Phabricator
      7. D13479.1.patch
        43 kB
        Phabricator

        Issue Links

          Activity

          Hide
          Owen O'Malley added a comment -

          I committed this to trunk and 0.13.

          Show
          Owen O'Malley added a comment - I committed this to trunk and 0.13.
          Hide
          Hive QA added a comment -

          Overall: +1 all checks pass

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12636771/HIVE-5567.patch

          SUCCESS: +1 5457 tests passed

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1962/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1962/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          

          This message is automatically generated.

          ATTACHMENT ID: 12636771

          Show
          Hive QA added a comment - Overall : +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12636771/HIVE-5567.patch SUCCESS: +1 5457 tests passed Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1962/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1962/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated. ATTACHMENT ID: 12636771
          Hide
          Alan Gates added a comment -

          Ran tests locally, all pass.

          Show
          Alan Gates added a comment - Ran tests locally, all pass.
          Hide
          Owen O'Malley added a comment -

          Fixed orc_analyze.q test to reflect the updated schema.

          Show
          Owen O'Malley added a comment - Fixed orc_analyze.q test to reflect the updated schema.
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12635678/HIVE-5567.patch

          ERROR: -1 due to 3 failed/errored test(s), 5440 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_analyze
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketmapjoin6
          org.apache.hive.service.cli.thrift.TestThriftBinaryCLIService.testExecuteStatementAsync
          

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1911/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1911/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 3 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12635678

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12635678/HIVE-5567.patch ERROR: -1 due to 3 failed/errored test(s), 5440 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_analyze org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketmapjoin6 org.apache.hive.service.cli.thrift.TestThriftBinaryCLIService.testExecuteStatementAsync Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1911/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1911/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 3 tests failed This message is automatically generated. ATTACHMENT ID: 12635678
          Hide
          Owen O'Malley added a comment -

          Reupload for jenkins. It compiles fine for me against trunk and the logs have timed out.

          Show
          Owen O'Malley added a comment - Reupload for jenkins. It compiles fine for me against trunk and the logs have timed out.
          Hide
          Hive QA added a comment -

          Overall: -1 no tests executed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12634784/HIVE-5567.patch

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1789/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1789/console

          Messages:

          **** This message was trimmed, see log for full details ****
          [INFO] Using 'UTF-8' encoding to copy filtered resources.
          [INFO] skip non existing resourceDirectory /data/hive-ptest/working/apache-svn-trunk-source/hwi/src/test/resources
          [INFO] Copying 3 resources
          [INFO] 
          [INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ hive-hwi ---
          [INFO] Executing tasks
          
          main:
              [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/hwi/target/tmp
              [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/hwi/target/warehouse
              [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/hwi/target/tmp/conf
               [copy] Copying 5 files to /data/hive-ptest/working/apache-svn-trunk-source/hwi/target/tmp/conf
          [INFO] Executed tasks
          [INFO] 
          [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ hive-hwi ---
          [INFO] Compiling 2 source files to /data/hive-ptest/working/apache-svn-trunk-source/hwi/target/test-classes
          [INFO] 
          [INFO] --- maven-surefire-plugin:2.16:test (default-test) @ hive-hwi ---
          [INFO] Tests are skipped.
          [INFO] 
          [INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ hive-hwi ---
          [INFO] Building jar: /data/hive-ptest/working/apache-svn-trunk-source/hwi/target/hive-hwi-0.14.0-SNAPSHOT.jar
          [INFO] 
          [INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @ hive-hwi ---
          [INFO] 
          [INFO] --- maven-install-plugin:2.4:install (default-install) @ hive-hwi ---
          [INFO] Installing /data/hive-ptest/working/apache-svn-trunk-source/hwi/target/hive-hwi-0.14.0-SNAPSHOT.jar to /data/hive-ptest/working/maven/org/apache/hive/hive-hwi/0.14.0-SNAPSHOT/hive-hwi-0.14.0-SNAPSHOT.jar
          [INFO] Installing /data/hive-ptest/working/apache-svn-trunk-source/hwi/pom.xml to /data/hive-ptest/working/maven/org/apache/hive/hive-hwi/0.14.0-SNAPSHOT/hive-hwi-0.14.0-SNAPSHOT.pom
          [INFO]                                                                         
          [INFO] ------------------------------------------------------------------------
          [INFO] Building Hive ODBC 0.14.0-SNAPSHOT
          [INFO] ------------------------------------------------------------------------
          [INFO] 
          [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-odbc ---
          [INFO] Deleting /data/hive-ptest/working/apache-svn-trunk-source/odbc (includes = [datanucleus.log, derby.log], excludes = [])
          [INFO] 
          [INFO] --- maven-remote-resources-plugin:1.5:process (default) @ hive-odbc ---
          [INFO] 
          [INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @ hive-odbc ---
          [INFO] Executing tasks
          
          main:
          [INFO] Executed tasks
          [INFO] 
          [INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ hive-odbc ---
          [INFO] Executing tasks
          
          main:
              [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/odbc/target/tmp
              [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/odbc/target/warehouse
              [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/odbc/target/tmp/conf
               [copy] Copying 5 files to /data/hive-ptest/working/apache-svn-trunk-source/odbc/target/tmp/conf
          [INFO] Executed tasks
          [INFO] 
          [INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @ hive-odbc ---
          [INFO] 
          [INFO] --- maven-install-plugin:2.4:install (default-install) @ hive-odbc ---
          [INFO] Installing /data/hive-ptest/working/apache-svn-trunk-source/odbc/pom.xml to /data/hive-ptest/working/maven/org/apache/hive/hive-odbc/0.14.0-SNAPSHOT/hive-odbc-0.14.0-SNAPSHOT.pom
          [INFO]                                                                         
          [INFO] ------------------------------------------------------------------------
          [INFO] Building Hive Shims Aggregator 0.14.0-SNAPSHOT
          [INFO] ------------------------------------------------------------------------
          [INFO] 
          [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-shims-aggregator ---
          [INFO] Deleting /data/hive-ptest/working/apache-svn-trunk-source/shims (includes = [datanucleus.log, derby.log], excludes = [])
          [INFO] 
          [INFO] --- maven-remote-resources-plugin:1.5:process (default) @ hive-shims-aggregator ---
          [INFO] 
          [INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @ hive-shims-aggregator ---
          [INFO] Executing tasks
          
          main:
          [INFO] Executed tasks
          [INFO] 
          [INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ hive-shims-aggregator ---
          [INFO] Executing tasks
          
          main:
              [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/shims/target/tmp
              [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/shims/target/warehouse
              [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/shims/target/tmp/conf
               [copy] Copying 5 files to /data/hive-ptest/working/apache-svn-trunk-source/shims/target/tmp/conf
          [INFO] Executed tasks
          [INFO] 
          [INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @ hive-shims-aggregator ---
          [INFO] 
          [INFO] --- maven-install-plugin:2.4:install (default-install) @ hive-shims-aggregator ---
          [INFO] Installing /data/hive-ptest/working/apache-svn-trunk-source/shims/pom.xml to /data/hive-ptest/working/maven/org/apache/hive/hive-shims-aggregator/0.14.0-SNAPSHOT/hive-shims-aggregator-0.14.0-SNAPSHOT.pom
          [INFO]                                                                         
          [INFO] ------------------------------------------------------------------------
          [INFO] Building Hive TestUtils 0.14.0-SNAPSHOT
          [INFO] ------------------------------------------------------------------------
          [INFO] 
          [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-testutils ---
          [INFO] Deleting /data/hive-ptest/working/apache-svn-trunk-source/testutils (includes = [datanucleus.log, derby.log], excludes = [])
          [INFO] 
          [INFO] --- maven-remote-resources-plugin:1.5:process (default) @ hive-testutils ---
          [INFO] 
          [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ hive-testutils ---
          [INFO] Using 'UTF-8' encoding to copy filtered resources.
          [INFO] skip non existing resourceDirectory /data/hive-ptest/working/apache-svn-trunk-source/testutils/src/main/resources
          [INFO] Copying 3 resources
          [INFO] 
          [INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @ hive-testutils ---
          [INFO] Executing tasks
          
          main:
          [INFO] Executed tasks
          [INFO] 
          [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hive-testutils ---
          [INFO] Compiling 2 source files to /data/hive-ptest/working/apache-svn-trunk-source/testutils/target/classes
          [INFO] 
          [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ hive-testutils ---
          [INFO] Using 'UTF-8' encoding to copy filtered resources.
          [INFO] skip non existing resourceDirectory /data/hive-ptest/working/apache-svn-trunk-source/testutils/src/test/resources
          [INFO] Copying 3 resources
          [INFO] 
          [INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ hive-testutils ---
          [INFO] Executing tasks
          
          main:
              [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/testutils/target/tmp
              [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/testutils/target/warehouse
              [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/testutils/target/tmp/conf
               [copy] Copying 5 files to /data/hive-ptest/working/apache-svn-trunk-source/testutils/target/tmp/conf
          [INFO] Executed tasks
          [INFO] 
          [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ hive-testutils ---
          [INFO] No sources to compile
          [INFO] 
          [INFO] --- maven-surefire-plugin:2.16:test (default-test) @ hive-testutils ---
          [INFO] Tests are skipped.
          [INFO] 
          [INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ hive-testutils ---
          [INFO] Building jar: /data/hive-ptest/working/apache-svn-trunk-source/testutils/target/hive-testutils-0.14.0-SNAPSHOT.jar
          [INFO] 
          [INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @ hive-testutils ---
          [INFO] 
          [INFO] --- maven-install-plugin:2.4:install (default-install) @ hive-testutils ---
          [INFO] Installing /data/hive-ptest/working/apache-svn-trunk-source/testutils/target/hive-testutils-0.14.0-SNAPSHOT.jar to /data/hive-ptest/working/maven/org/apache/hive/hive-testutils/0.14.0-SNAPSHOT/hive-testutils-0.14.0-SNAPSHOT.jar
          [INFO] Installing /data/hive-ptest/working/apache-svn-trunk-source/testutils/pom.xml to /data/hive-ptest/working/maven/org/apache/hive/hive-testutils/0.14.0-SNAPSHOT/hive-testutils-0.14.0-SNAPSHOT.pom
          [INFO]                                                                         
          [INFO] ------------------------------------------------------------------------
          [INFO] Building Hive Packaging 0.14.0-SNAPSHOT
          [INFO] ------------------------------------------------------------------------
          Downloading: http://repository.apache.org/snapshots/org/apache/hive/hcatalog/hive-hcatalog-hbase-storage-handler/0.14.0-SNAPSHOT/maven-metadata.xml
          Downloading: http://repository.apache.org/snapshots/org/apache/hive/hcatalog/hive-hcatalog-hbase-storage-handler/0.14.0-SNAPSHOT/hive-hcatalog-hbase-storage-handler-0.14.0-SNAPSHOT.pom
          [WARNING] The POM for org.apache.hive.hcatalog:hive-hcatalog-hbase-storage-handler:jar:0.14.0-SNAPSHOT is missing, no dependency information available
          Downloading: http://repository.apache.org/snapshots/org/apache/hive/hcatalog/hive-hcatalog-hbase-storage-handler/0.14.0-SNAPSHOT/hive-hcatalog-hbase-storage-handler-0.14.0-SNAPSHOT.jar
          [INFO] ------------------------------------------------------------------------
          [INFO] Reactor Summary:
          [INFO] 
          [INFO] Hive .............................................. SUCCESS [8.711s]
          [INFO] Hive Ant Utilities ................................ SUCCESS [5.610s]
          [INFO] Hive Shims Common ................................. SUCCESS [3.784s]
          [INFO] Hive Shims 0.20 ................................... SUCCESS [2.541s]
          [INFO] Hive Shims Secure Common .......................... SUCCESS [4.534s]
          [INFO] Hive Shims 0.20S .................................. SUCCESS [3.049s]
          [INFO] Hive Shims 0.23 ................................... SUCCESS [7.939s]
          [INFO] Hive Shims ........................................ SUCCESS [1.269s]
          [INFO] Hive Common ....................................... SUCCESS [6.541s]
          [INFO] Hive Serde ........................................ SUCCESS [9.768s]
          [INFO] Hive Metastore .................................... SUCCESS [32.340s]
          [INFO] Hive Query Language ............................... SUCCESS [1:18.562s]
          [INFO] Hive Service ...................................... SUCCESS [8.103s]
          [INFO] Hive JDBC ......................................... SUCCESS [2.930s]
          [INFO] Hive Beeline ...................................... SUCCESS [3.400s]
          [INFO] Hive CLI .......................................... SUCCESS [2.617s]
          [INFO] Hive Contrib ...................................... SUCCESS [2.768s]
          [INFO] Hive HBase Handler ................................ SUCCESS [2.655s]
          [INFO] Hive HCatalog ..................................... SUCCESS [0.549s]
          [INFO] Hive HCatalog Core ................................ SUCCESS [2.009s]
          [INFO] Hive HCatalog Pig Adapter ......................... SUCCESS [2.447s]
          [INFO] Hive HCatalog Server Extensions ................... SUCCESS [1.194s]
          [INFO] Hive HCatalog Webhcat Java Client ................. SUCCESS [2.098s]
          [INFO] Hive HCatalog Webhcat ............................. SUCCESS [9.913s]
          [INFO] Hive HWI .......................................... SUCCESS [1.081s]
          [INFO] Hive ODBC ......................................... SUCCESS [0.786s]
          [INFO] Hive Shims Aggregator ............................. SUCCESS [0.282s]
          [INFO] Hive TestUtils .................................... SUCCESS [0.731s]
          [INFO] Hive Packaging .................................... FAILURE [2.377s]
          [INFO] ------------------------------------------------------------------------
          [INFO] BUILD FAILURE
          [INFO] ------------------------------------------------------------------------
          [INFO] Total time: 3:35.120s
          [INFO] Finished at: Sat Mar 15 02:59:04 EDT 2014
          [INFO] Final Memory: 74M/425M
          [INFO] ------------------------------------------------------------------------
          [ERROR] Failed to execute goal on project hive-packaging: Could not resolve dependencies for project org.apache.hive:hive-packaging:pom:0.14.0-SNAPSHOT: Could not find artifact org.apache.hive.hcatalog:hive-hcatalog-hbase-storage-handler:jar:0.14.0-SNAPSHOT in apache.snapshots (http://repository.apache.org/snapshots) -> [Help 1]
          [ERROR] 
          [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
          [ERROR] Re-run Maven using the -X switch to enable full debug logging.
          [ERROR] 
          [ERROR] For more information about the errors and possible solutions, please read the following articles:
          [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
          [ERROR] 
          [ERROR] After correcting the problems, you can resume the build with the command
          [ERROR]   mvn <goals> -rf :hive-packaging
          + exit 1
          '
          

          This message is automatically generated.

          ATTACHMENT ID: 12634784

          Show
          Hive QA added a comment - Overall : -1 no tests executed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12634784/HIVE-5567.patch Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1789/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1789/console Messages: **** This message was trimmed, see log for full details **** [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /data/hive-ptest/working/apache-svn-trunk-source/hwi/src/test/resources [INFO] Copying 3 resources [INFO] [INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ hive-hwi --- [INFO] Executing tasks main: [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/hwi/target/tmp [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/hwi/target/warehouse [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/hwi/target/tmp/conf [copy] Copying 5 files to /data/hive-ptest/working/apache-svn-trunk-source/hwi/target/tmp/conf [INFO] Executed tasks [INFO] [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ hive-hwi --- [INFO] Compiling 2 source files to /data/hive-ptest/working/apache-svn-trunk-source/hwi/target/test-classes [INFO] [INFO] --- maven-surefire-plugin:2.16:test (default-test) @ hive-hwi --- [INFO] Tests are skipped. [INFO] [INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ hive-hwi --- [INFO] Building jar: /data/hive-ptest/working/apache-svn-trunk-source/hwi/target/hive-hwi-0.14.0-SNAPSHOT.jar [INFO] [INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @ hive-hwi --- [INFO] [INFO] --- maven-install-plugin:2.4:install (default-install) @ hive-hwi --- [INFO] Installing /data/hive-ptest/working/apache-svn-trunk-source/hwi/target/hive-hwi-0.14.0-SNAPSHOT.jar to /data/hive-ptest/working/maven/org/apache/hive/hive-hwi/0.14.0-SNAPSHOT/hive-hwi-0.14.0-SNAPSHOT.jar [INFO] Installing /data/hive-ptest/working/apache-svn-trunk-source/hwi/pom.xml to /data/hive-ptest/working/maven/org/apache/hive/hive-hwi/0.14.0-SNAPSHOT/hive-hwi-0.14.0-SNAPSHOT.pom [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building Hive ODBC 0.14.0-SNAPSHOT [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-odbc --- [INFO] Deleting /data/hive-ptest/working/apache-svn-trunk-source/odbc (includes = [datanucleus.log, derby.log], excludes = []) [INFO] [INFO] --- maven-remote-resources-plugin:1.5:process (default) @ hive-odbc --- [INFO] [INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @ hive-odbc --- [INFO] Executing tasks main: [INFO] Executed tasks [INFO] [INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ hive-odbc --- [INFO] Executing tasks main: [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/odbc/target/tmp [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/odbc/target/warehouse [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/odbc/target/tmp/conf [copy] Copying 5 files to /data/hive-ptest/working/apache-svn-trunk-source/odbc/target/tmp/conf [INFO] Executed tasks [INFO] [INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @ hive-odbc --- [INFO] [INFO] --- maven-install-plugin:2.4:install (default-install) @ hive-odbc --- [INFO] Installing /data/hive-ptest/working/apache-svn-trunk-source/odbc/pom.xml to /data/hive-ptest/working/maven/org/apache/hive/hive-odbc/0.14.0-SNAPSHOT/hive-odbc-0.14.0-SNAPSHOT.pom [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building Hive Shims Aggregator 0.14.0-SNAPSHOT [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-shims-aggregator --- [INFO] Deleting /data/hive-ptest/working/apache-svn-trunk-source/shims (includes = [datanucleus.log, derby.log], excludes = []) [INFO] [INFO] --- maven-remote-resources-plugin:1.5:process (default) @ hive-shims-aggregator --- [INFO] [INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @ hive-shims-aggregator --- [INFO] Executing tasks main: [INFO] Executed tasks [INFO] [INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ hive-shims-aggregator --- [INFO] Executing tasks main: [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/shims/target/tmp [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/shims/target/warehouse [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/shims/target/tmp/conf [copy] Copying 5 files to /data/hive-ptest/working/apache-svn-trunk-source/shims/target/tmp/conf [INFO] Executed tasks [INFO] [INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @ hive-shims-aggregator --- [INFO] [INFO] --- maven-install-plugin:2.4:install (default-install) @ hive-shims-aggregator --- [INFO] Installing /data/hive-ptest/working/apache-svn-trunk-source/shims/pom.xml to /data/hive-ptest/working/maven/org/apache/hive/hive-shims-aggregator/0.14.0-SNAPSHOT/hive-shims-aggregator-0.14.0-SNAPSHOT.pom [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building Hive TestUtils 0.14.0-SNAPSHOT [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-testutils --- [INFO] Deleting /data/hive-ptest/working/apache-svn-trunk-source/testutils (includes = [datanucleus.log, derby.log], excludes = []) [INFO] [INFO] --- maven-remote-resources-plugin:1.5:process (default) @ hive-testutils --- [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ hive-testutils --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /data/hive-ptest/working/apache-svn-trunk-source/testutils/src/main/resources [INFO] Copying 3 resources [INFO] [INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @ hive-testutils --- [INFO] Executing tasks main: [INFO] Executed tasks [INFO] [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hive-testutils --- [INFO] Compiling 2 source files to /data/hive-ptest/working/apache-svn-trunk-source/testutils/target/classes [INFO] [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ hive-testutils --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /data/hive-ptest/working/apache-svn-trunk-source/testutils/src/test/resources [INFO] Copying 3 resources [INFO] [INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ hive-testutils --- [INFO] Executing tasks main: [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/testutils/target/tmp [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/testutils/target/warehouse [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/testutils/target/tmp/conf [copy] Copying 5 files to /data/hive-ptest/working/apache-svn-trunk-source/testutils/target/tmp/conf [INFO] Executed tasks [INFO] [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ hive-testutils --- [INFO] No sources to compile [INFO] [INFO] --- maven-surefire-plugin:2.16:test (default-test) @ hive-testutils --- [INFO] Tests are skipped. [INFO] [INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ hive-testutils --- [INFO] Building jar: /data/hive-ptest/working/apache-svn-trunk-source/testutils/target/hive-testutils-0.14.0-SNAPSHOT.jar [INFO] [INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @ hive-testutils --- [INFO] [INFO] --- maven-install-plugin:2.4:install (default-install) @ hive-testutils --- [INFO] Installing /data/hive-ptest/working/apache-svn-trunk-source/testutils/target/hive-testutils-0.14.0-SNAPSHOT.jar to /data/hive-ptest/working/maven/org/apache/hive/hive-testutils/0.14.0-SNAPSHOT/hive-testutils-0.14.0-SNAPSHOT.jar [INFO] Installing /data/hive-ptest/working/apache-svn-trunk-source/testutils/pom.xml to /data/hive-ptest/working/maven/org/apache/hive/hive-testutils/0.14.0-SNAPSHOT/hive-testutils-0.14.0-SNAPSHOT.pom [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building Hive Packaging 0.14.0-SNAPSHOT [INFO] ------------------------------------------------------------------------ Downloading: http://repository.apache.org/snapshots/org/apache/hive/hcatalog/hive-hcatalog-hbase-storage-handler/0.14.0-SNAPSHOT/maven-metadata.xml Downloading: http://repository.apache.org/snapshots/org/apache/hive/hcatalog/hive-hcatalog-hbase-storage-handler/0.14.0-SNAPSHOT/hive-hcatalog-hbase-storage-handler-0.14.0-SNAPSHOT.pom [WARNING] The POM for org.apache.hive.hcatalog:hive-hcatalog-hbase-storage-handler:jar:0.14.0-SNAPSHOT is missing, no dependency information available Downloading: http://repository.apache.org/snapshots/org/apache/hive/hcatalog/hive-hcatalog-hbase-storage-handler/0.14.0-SNAPSHOT/hive-hcatalog-hbase-storage-handler-0.14.0-SNAPSHOT.jar [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Hive .............................................. SUCCESS [8.711s] [INFO] Hive Ant Utilities ................................ SUCCESS [5.610s] [INFO] Hive Shims Common ................................. SUCCESS [3.784s] [INFO] Hive Shims 0.20 ................................... SUCCESS [2.541s] [INFO] Hive Shims Secure Common .......................... SUCCESS [4.534s] [INFO] Hive Shims 0.20S .................................. SUCCESS [3.049s] [INFO] Hive Shims 0.23 ................................... SUCCESS [7.939s] [INFO] Hive Shims ........................................ SUCCESS [1.269s] [INFO] Hive Common ....................................... SUCCESS [6.541s] [INFO] Hive Serde ........................................ SUCCESS [9.768s] [INFO] Hive Metastore .................................... SUCCESS [32.340s] [INFO] Hive Query Language ............................... SUCCESS [1:18.562s] [INFO] Hive Service ...................................... SUCCESS [8.103s] [INFO] Hive JDBC ......................................... SUCCESS [2.930s] [INFO] Hive Beeline ...................................... SUCCESS [3.400s] [INFO] Hive CLI .......................................... SUCCESS [2.617s] [INFO] Hive Contrib ...................................... SUCCESS [2.768s] [INFO] Hive HBase Handler ................................ SUCCESS [2.655s] [INFO] Hive HCatalog ..................................... SUCCESS [0.549s] [INFO] Hive HCatalog Core ................................ SUCCESS [2.009s] [INFO] Hive HCatalog Pig Adapter ......................... SUCCESS [2.447s] [INFO] Hive HCatalog Server Extensions ................... SUCCESS [1.194s] [INFO] Hive HCatalog Webhcat Java Client ................. SUCCESS [2.098s] [INFO] Hive HCatalog Webhcat ............................. SUCCESS [9.913s] [INFO] Hive HWI .......................................... SUCCESS [1.081s] [INFO] Hive ODBC ......................................... SUCCESS [0.786s] [INFO] Hive Shims Aggregator ............................. SUCCESS [0.282s] [INFO] Hive TestUtils .................................... SUCCESS [0.731s] [INFO] Hive Packaging .................................... FAILURE [2.377s] [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 3:35.120s [INFO] Finished at: Sat Mar 15 02:59:04 EDT 2014 [INFO] Final Memory: 74M/425M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal on project hive-packaging: Could not resolve dependencies for project org.apache.hive:hive-packaging:pom:0.14.0-SNAPSHOT: Could not find artifact org.apache.hive.hcatalog:hive-hcatalog-hbase-storage-handler:jar:0.14.0-SNAPSHOT in apache.snapshots (http://repository.apache.org/snapshots) -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn <goals> -rf :hive-packaging + exit 1 ' This message is automatically generated. ATTACHMENT ID: 12634784
          Hide
          Owen O'Malley added a comment -

          Reupload for jenkins.

          Show
          Owen O'Malley added a comment - Reupload for jenkins.
          Hide
          Gunther Hagleitner added a comment -

          looked at it again. still looks good. +1

          Show
          Gunther Hagleitner added a comment - looked at it again. still looks good. +1
          Hide
          Owen O'Malley added a comment -

          I've rebased the original test and added a new test for timestamp.

          Show
          Owen O'Malley added a comment - I've rebased the original test and added a new test for timestamp.
          Hide
          Hive QA added a comment -

          Overall: -1 no tests executed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12609023/D13479.3.patch

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1462/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1462/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Tests exited with: NonZeroExitCodeException
          Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n '' ]]
          + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
          + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
          + export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
          + M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
          + cd /data/hive-ptest/working/
          + tee /data/hive-ptest/logs/PreCommit-HIVE-Build-1462/source-prep.txt
          + [[ false == \t\r\u\e ]]
          + mkdir -p maven ivy
          + [[ svn = \s\v\n ]]
          + [[ -n '' ]]
          + [[ -d apache-svn-trunk-source ]]
          + [[ ! -d apache-svn-trunk-source/.svn ]]
          + [[ ! -d apache-svn-trunk-source ]]
          + cd apache-svn-trunk-source
          + svn revert -R .
          Reverted 'pom.xml'
          Reverted 'common/src/java/org/apache/hadoop/hive/conf/HiveConf.java'
          Reverted 'service/src/java/org/apache/hive/service/auth/AuthenticationProviderFactory.java'
          Reverted 'service/pom.xml'
          ++ awk '{print $2}'
          ++ egrep -v '^X|^Performing status on external'
          ++ svn status --no-ignore
          + rm -rf target datanucleus.log ant/target shims/target shims/0.20/target shims/0.20S/target shims/0.23/target shims/aggregator/target shims/common/target shims/common-secure/target packaging/target hbase-handler/target testutils/target jdbc/target metastore/target itests/target itests/hcatalog-unit/target itests/test-serde/target itests/qtest/target itests/hive-unit/target itests/custom-serde/target itests/util/target hcatalog/target hcatalog/storage-handlers/hbase/target hcatalog/server-extensions/target hcatalog/core/target hcatalog/webhcat/svr/target hcatalog/webhcat/java-client/target hcatalog/hcatalog-pig-adapter/target hwi/target common/target common/src/gen common/src/java/org/apache/hadoop/hive/conf/HiveConf.java.orig contrib/target service/target service/src/java/org/apache/hive/service/auth/PamAuthenticationProviderImpl.java serde/target beeline/target odbc/target cli/target ql/dependency-reduced-pom.xml ql/target
          + svn update
          
          Fetching external item into 'hcatalog/src/test/e2e/harness'
          External at revision 1571112.
          
          At revision 1571112.
          + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
          + patchFilePath=/data/hive-ptest/working/scratch/build.patch
          + [[ -f /data/hive-ptest/working/scratch/build.patch ]]
          + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
          + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch
          The patch does not appear to apply with p0, p1, or p2
          + exit 1
          '
          

          This message is automatically generated.

          ATTACHMENT ID: 12609023

          Show
          Hive QA added a comment - Overall : -1 no tests executed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12609023/D13479.3.patch Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1462/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1462/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n '' ]] + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + cd /data/hive-ptest/working/ + tee /data/hive-ptest/logs/PreCommit-HIVE-Build-1462/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ svn = \s\v\n ]] + [[ -n '' ]] + [[ -d apache-svn-trunk-source ]] + [[ ! -d apache-svn-trunk-source/.svn ]] + [[ ! -d apache-svn-trunk-source ]] + cd apache-svn-trunk-source + svn revert -R . Reverted 'pom.xml' Reverted 'common/src/java/org/apache/hadoop/hive/conf/HiveConf.java' Reverted 'service/src/java/org/apache/hive/service/auth/AuthenticationProviderFactory.java' Reverted 'service/pom.xml' ++ awk '{print $2}' ++ egrep -v '^X|^Performing status on external' ++ svn status --no-ignore + rm -rf target datanucleus.log ant/target shims/target shims/0.20/target shims/0.20S/target shims/0.23/target shims/aggregator/target shims/common/target shims/common-secure/target packaging/target hbase-handler/target testutils/target jdbc/target metastore/target itests/target itests/hcatalog-unit/target itests/test-serde/target itests/qtest/target itests/hive-unit/target itests/custom-serde/target itests/util/target hcatalog/target hcatalog/storage-handlers/hbase/target hcatalog/server-extensions/target hcatalog/core/target hcatalog/webhcat/svr/target hcatalog/webhcat/java-client/target hcatalog/hcatalog-pig-adapter/target hwi/target common/target common/src/gen common/src/java/org/apache/hadoop/hive/conf/HiveConf.java.orig contrib/target service/target service/src/java/org/apache/hive/service/auth/PamAuthenticationProviderImpl.java serde/target beeline/target odbc/target cli/target ql/dependency-reduced-pom.xml ql/target + svn update Fetching external item into 'hcatalog/src/test/e2e/harness' External at revision 1571112. At revision 1571112. + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hive-ptest/working/scratch/build.patch + [[ -f /data/hive-ptest/working/scratch/build.patch ]] + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch The patch does not appear to apply with p0, p1, or p2 + exit 1 ' This message is automatically generated. ATTACHMENT ID: 12609023
          Hide
          Hive QA added a comment -

          Overall: +1 all checks pass

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12609023/D13479.3.patch

          SUCCESS: +1 4427 tests passed

          Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1166/testReport
          Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1166/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          

          This message is automatically generated.

          Show
          Hive QA added a comment - Overall : +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12609023/D13479.3.patch SUCCESS: +1 4427 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1166/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1166/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated.
          Hide
          Phabricator added a comment -

          hagleitn has commented on the revision "HIVE-5567 [jira] Add better protection code for SARGs".

          LGTM

          REVISION DETAIL
          https://reviews.facebook.net/D13479

          BRANCH
          h-5567

          ARCANIST PROJECT
          hive

          To: JIRA, hagleitn, omalley

          Show
          Phabricator added a comment - hagleitn has commented on the revision " HIVE-5567 [jira] Add better protection code for SARGs". LGTM REVISION DETAIL https://reviews.facebook.net/D13479 BRANCH h-5567 ARCANIST PROJECT hive To: JIRA, hagleitn, omalley
          Hide
          Phabricator added a comment -

          omalley updated the revision "HIVE-5567 [jira] Add better protection code for SARGs".

          Fixed the ordering to fix TestSearchArgumentImpl

          Reviewers: hagleitn, JIRA

          REVISION DETAIL
          https://reviews.facebook.net/D13479

          CHANGE SINCE LAST DIFF
          https://reviews.facebook.net/D13479?vs=41685&id=41781#toc

          BRANCH
          h-5567

          ARCANIST PROJECT
          hive

          AFFECTED FILES
          data/files/orc_create_people.txt
          ql/src/java/org/apache/hadoop/hive/ql/io/sarg/SearchArgumentImpl.java
          ql/src/test/queries/clientpositive/orc_create.q
          ql/src/test/results/clientpositive/orc_create.q.out

          To: JIRA, hagleitn, omalley

          Show
          Phabricator added a comment - omalley updated the revision " HIVE-5567 [jira] Add better protection code for SARGs". Fixed the ordering to fix TestSearchArgumentImpl Reviewers: hagleitn, JIRA REVISION DETAIL https://reviews.facebook.net/D13479 CHANGE SINCE LAST DIFF https://reviews.facebook.net/D13479?vs=41685&id=41781#toc BRANCH h-5567 ARCANIST PROJECT hive AFFECTED FILES data/files/orc_create_people.txt ql/src/java/org/apache/hadoop/hive/ql/io/sarg/SearchArgumentImpl.java ql/src/test/queries/clientpositive/orc_create.q ql/src/test/results/clientpositive/orc_create.q.out To: JIRA, hagleitn, omalley
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12608836/D13479.2.patch

          ERROR: -1 due to 5 failed/errored test(s), 4411 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_pushdown
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_script_broken_pipe1
          org.apache.hadoop.hive.ql.io.sarg.TestSearchArgumentImpl.testExpression3
          org.apache.hadoop.hive.ql.io.sarg.TestSearchArgumentImpl.testExpression5
          org.apache.hadoop.hive.ql.io.sarg.TestSearchArgumentImpl.testExpression8
          

          Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1150/testReport
          Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1150/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests failed with: TestsFailedException: 5 tests failed
          

          This message is automatically generated.

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12608836/D13479.2.patch ERROR: -1 due to 5 failed/errored test(s), 4411 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_pushdown org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_script_broken_pipe1 org.apache.hadoop.hive.ql.io.sarg.TestSearchArgumentImpl.testExpression3 org.apache.hadoop.hive.ql.io.sarg.TestSearchArgumentImpl.testExpression5 org.apache.hadoop.hive.ql.io.sarg.TestSearchArgumentImpl.testExpression8 Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1150/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1150/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests failed with: TestsFailedException: 5 tests failed This message is automatically generated.
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12608836/D13479.2.patch

          ERROR: -1 due to 4 failed/errored test(s), 4411 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_pushdown
          org.apache.hadoop.hive.ql.io.sarg.TestSearchArgumentImpl.testExpression3
          org.apache.hadoop.hive.ql.io.sarg.TestSearchArgumentImpl.testExpression5
          org.apache.hadoop.hive.ql.io.sarg.TestSearchArgumentImpl.testExpression8
          

          Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1149/testReport
          Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1149/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests failed with: TestsFailedException: 4 tests failed
          

          This message is automatically generated.

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12608836/D13479.2.patch ERROR: -1 due to 4 failed/errored test(s), 4411 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_pushdown org.apache.hadoop.hive.ql.io.sarg.TestSearchArgumentImpl.testExpression3 org.apache.hadoop.hive.ql.io.sarg.TestSearchArgumentImpl.testExpression5 org.apache.hadoop.hive.ql.io.sarg.TestSearchArgumentImpl.testExpression8 Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1149/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1149/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests failed with: TestsFailedException: 4 tests failed This message is automatically generated.
          Hide
          Phabricator added a comment -

          omalley updated the revision "HIVE-5567 [jira] Add better protection code for SARGs".

          Fixed the test case to keep the int column

          Reviewers: hagleitn, JIRA

          REVISION DETAIL
          https://reviews.facebook.net/D13479

          CHANGE SINCE LAST DIFF
          https://reviews.facebook.net/D13479?vs=41637&id=41685#toc

          BRANCH
          h-5567

          ARCANIST PROJECT
          hive

          AFFECTED FILES
          data/files/orc_create_people.txt
          ql/src/java/org/apache/hadoop/hive/ql/io/sarg/SearchArgumentImpl.java
          ql/src/test/queries/clientpositive/orc_create.q
          ql/src/test/results/clientpositive/orc_create.q.out

          To: JIRA, hagleitn, omalley

          Show
          Phabricator added a comment - omalley updated the revision " HIVE-5567 [jira] Add better protection code for SARGs". Fixed the test case to keep the int column Reviewers: hagleitn, JIRA REVISION DETAIL https://reviews.facebook.net/D13479 CHANGE SINCE LAST DIFF https://reviews.facebook.net/D13479?vs=41637&id=41685#toc BRANCH h-5567 ARCANIST PROJECT hive AFFECTED FILES data/files/orc_create_people.txt ql/src/java/org/apache/hadoop/hive/ql/io/sarg/SearchArgumentImpl.java ql/src/test/queries/clientpositive/orc_create.q ql/src/test/results/clientpositive/orc_create.q.out To: JIRA, hagleitn, omalley
          Hide
          Gunther Hagleitner added a comment -

          +1 pending the small test case update

          Show
          Gunther Hagleitner added a comment - +1 pending the small test case update
          Hide
          Phabricator added a comment -

          hagleitn has accepted the revision "HIVE-5567 [jira] Add better protection code for SARGs".

          Looks good except for minor test issue.

          INLINE COMMENTS
          ql/src/test/queries/clientpositive/orc_create.q:69 did you mean to change this to decimal? i'm guessing you want an int test too.

          REVISION DETAIL
          https://reviews.facebook.net/D13479

          BRANCH
          h-5567

          ARCANIST PROJECT
          hive

          To: JIRA, hagleitn, omalley

          Show
          Phabricator added a comment - hagleitn has accepted the revision " HIVE-5567 [jira] Add better protection code for SARGs". Looks good except for minor test issue. INLINE COMMENTS ql/src/test/queries/clientpositive/orc_create.q:69 did you mean to change this to decimal? i'm guessing you want an int test too. REVISION DETAIL https://reviews.facebook.net/D13479 BRANCH h-5567 ARCANIST PROJECT hive To: JIRA, hagleitn, omalley
          Hide
          Phabricator added a comment -

          omalley requested code review of "HIVE-5567 [jira] Add better protection code for SARGs".

          Reviewers: JIRA

          HIVE-5567. Add better protection code for SARGs.

          Currently, the SARG parser gets a NPE when the push down predicate uses a type like decimal that isn't supported.

          TEST PLAN
          EMPTY

          REVISION DETAIL
          https://reviews.facebook.net/D13479

          AFFECTED FILES
          data/files/orc_create_people.txt
          ql/src/java/org/apache/hadoop/hive/ql/io/sarg/SearchArgumentImpl.java
          ql/src/test/queries/clientpositive/orc_create.q
          ql/src/test/results/clientpositive/orc_create.q.out

          MANAGE HERALD RULES
          https://reviews.facebook.net/herald/view/differential/

          WHY DID I GET THIS EMAIL?
          https://reviews.facebook.net/herald/transcript/40305/

          To: JIRA, omalley

          Show
          Phabricator added a comment - omalley requested code review of " HIVE-5567 [jira] Add better protection code for SARGs". Reviewers: JIRA HIVE-5567 . Add better protection code for SARGs. Currently, the SARG parser gets a NPE when the push down predicate uses a type like decimal that isn't supported. TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D13479 AFFECTED FILES data/files/orc_create_people.txt ql/src/java/org/apache/hadoop/hive/ql/io/sarg/SearchArgumentImpl.java ql/src/test/queries/clientpositive/orc_create.q ql/src/test/results/clientpositive/orc_create.q.out MANAGE HERALD RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/40305/ To: JIRA, omalley
          Hide
          Owen O'Malley added a comment -

          The SARG is a best effort for optimization. It doesn't guarantee that all types and predicates are supported. For example, if you do a UDF like "where MyUdf(X) = 20" that can't be translated in to a SARG. Even for predicates that can be handled, currently only blocks of rows are tested and are accepted if any of the rows may pass the complete filter. In all cases the complete filter is still applied by Hive. SARGs are just optimizing which groups of rows need to be read from HDFS at all.

          That said, we should add decimal, date, and timestamp support to SARGs. That will be a bigger project and I'll file a separate jira.

          This issue is about preventing the optimization from causing run time errors. smile

          Show
          Owen O'Malley added a comment - The SARG is a best effort for optimization. It doesn't guarantee that all types and predicates are supported. For example, if you do a UDF like "where MyUdf(X) = 20" that can't be translated in to a SARG. Even for predicates that can be handled, currently only blocks of rows are tested and are accepted if any of the rows may pass the complete filter. In all cases the complete filter is still applied by Hive. SARGs are just optimizing which groups of rows need to be read from HDFS at all. That said, we should add decimal, date, and timestamp support to SARGs. That will be a bigger project and I'll file a separate jira. This issue is about preventing the optimization from causing run time errors. smile
          Hide
          Owen O'Malley added a comment -

          The exception looks like:

          Caused by: java.lang.NullPointerException
                  at org.apache.hadoop.hive.ql.io.sarg.SearchArgumentImpl$PredicateLeafImpl.hashCode(SearchArgumentImpl.java:148)
                  at java.util.HashMap.get(HashMap.java:300)
                  at org.apache.hadoop.hive.ql.io.sarg.SearchArgumentImpl$ExpressionBuilder.buildLeafList(SearchArgumentImpl.java:740)
                  at org.apache.hadoop.hive.ql.io.sarg.SearchArgumentImpl$ExpressionBuilder.buildLeafList(SearchArgumentImpl.java:735)
                  at org.apache.hadoop.hive.ql.io.sarg.SearchArgumentImpl$ExpressionBuilder.expression(SearchArgumentImpl.java:776)
                  at org.apache.hadoop.hive.ql.io.sarg.SearchArgumentImpl$ExpressionBuilder.expression(SearchArgumentImpl.java:759)
                  at org.apache.hadoop.hive.ql.io.sarg.SearchArgumentImpl.<init>(SearchArgumentImpl.java:795)
                  at org.apache.hadoop.hive.ql.io.sarg.SearchArgument$Factory.create(SearchArgument.java:174)
                  at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.createSarg(OrcInputFormat.java:172)
                  at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$OrcRecordReader.<init>(OrcInputFormat.java:98)
                  at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:241)
                  at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.<init>(CombineHiveRecordReader.java:65)
                  ... 12 more
          
          Show
          Owen O'Malley added a comment - The exception looks like: Caused by: java.lang.NullPointerException at org.apache.hadoop.hive.ql.io.sarg.SearchArgumentImpl$PredicateLeafImpl.hashCode(SearchArgumentImpl.java:148) at java.util.HashMap.get(HashMap.java:300) at org.apache.hadoop.hive.ql.io.sarg.SearchArgumentImpl$ExpressionBuilder.buildLeafList(SearchArgumentImpl.java:740) at org.apache.hadoop.hive.ql.io.sarg.SearchArgumentImpl$ExpressionBuilder.buildLeafList(SearchArgumentImpl.java:735) at org.apache.hadoop.hive.ql.io.sarg.SearchArgumentImpl$ExpressionBuilder.expression(SearchArgumentImpl.java:776) at org.apache.hadoop.hive.ql.io.sarg.SearchArgumentImpl$ExpressionBuilder.expression(SearchArgumentImpl.java:759) at org.apache.hadoop.hive.ql.io.sarg.SearchArgumentImpl.<init>(SearchArgumentImpl.java:795) at org.apache.hadoop.hive.ql.io.sarg.SearchArgument$Factory.create(SearchArgument.java:174) at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.createSarg(OrcInputFormat.java:172) at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$OrcRecordReader.<init>(OrcInputFormat.java:98) at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:241) at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.<init>(CombineHiveRecordReader.java:65) ... 12 more
          Hide
          Edward Capriolo added a comment -

          In other words we do not want to create fragmentation. If types certain types can not work with predicate-pushdown that is a problem we should address.

          Show
          Edward Capriolo added a comment - In other words we do not want to create fragmentation. If types certain types can not work with predicate-pushdown that is a problem we should address.
          Hide
          Edward Capriolo added a comment -

          Is there a reason that decimal can not be supported, or is the support for decimal incomplete?

          If SARG can support decimal we might be better off not adding protection, instead we should ensure that our unit tests cover all types.

          Show
          Edward Capriolo added a comment - Is there a reason that decimal can not be supported, or is the support for decimal incomplete? If SARG can support decimal we might be better off not adding protection, instead we should ensure that our unit tests cover all types.

            People

            • Assignee:
              Owen O'Malley
              Reporter:
              Owen O'Malley
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development