Hive
  1. Hive
  2. HIVE-7849

Support more generic predicate pushdown for hbase handler

    Details

    • Type: Improvement Improvement
    • Status: Patch Available
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: HBase Handler
    • Labels:
      None

      Description

      Currently, hbase handler supports AND conjugated filters only. This is the first try to support OR, NOT, IN, BETWEEN predicates for hbase.
      Mostly based on the work done by Teddy Choi.

      1. HIVE-7849.1.patch.txt
        118 kB
        Navis
      2. HIVE-7849.2.patch.txt
        142 kB
        Navis

        Issue Links

          Activity

          Hide
          Navis added a comment -

          Running preliminary test. Need some more elaboration on interfaces, etc.

          Show
          Navis added a comment - Running preliminary test. Need some more elaboration on interfaces, etc.
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12663627/HIVE-7849.1.patch.txt

          ERROR: -1 due to 8 failed/errored test(s), 6117 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_between
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_between_in
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_opt_vectorization
          org.apache.hadoop.hive.metastore.TestMetastoreExpr.testPartitionExpr
          org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapRedPlan6
          org.apache.hadoop.hive.ql.exec.vector.TestVectorizationContext.testBetweenFilters
          org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler.testPigFilterProjection
          org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/464/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/464/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-464/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 8 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12663627

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12663627/HIVE-7849.1.patch.txt ERROR: -1 due to 8 failed/errored test(s), 6117 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_between org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_between_in org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_opt_vectorization org.apache.hadoop.hive.metastore.TestMetastoreExpr.testPartitionExpr org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapRedPlan6 org.apache.hadoop.hive.ql.exec.vector.TestVectorizationContext.testBetweenFilters org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler.testPigFilterProjection org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/464/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/464/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-464/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 8 tests failed This message is automatically generated. ATTACHMENT ID: 12663627
          Hide
          Andrew Mains added a comment -

          Awesome feature! This would be very useful for HIVE-7805--given the two tickets, one could split up an IN condition on a composite key field into multiple scans.

          Show
          Andrew Mains added a comment - Awesome feature! This would be very useful for HIVE-7805 --given the two tickets, one could split up an IN condition on a composite key field into multiple scans.
          Hide
          Navis added a comment -

          Fixed test fails

          Show
          Navis added a comment - Fixed test fails
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12664070/HIVE-7849.2.patch.txt

          ERROR: -1 due to 6 failed/errored test(s), 6119 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_between_in
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_opt_vectorization
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority
          org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler.testPigFilterProjection
          org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataPrimitiveTypes
          org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/481/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/481/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-481/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 6 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12664070

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12664070/HIVE-7849.2.patch.txt ERROR: -1 due to 6 failed/errored test(s), 6119 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_between_in org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_opt_vectorization org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler.testPigFilterProjection org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataPrimitiveTypes org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/481/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/481/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-481/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 6 tests failed This message is automatically generated. ATTACHMENT ID: 12664070
          Hide
          Navis added a comment -

          vector_between_in need update. Others are not related to this.

          Show
          Navis added a comment - vector_between_in need update. Others are not related to this.

            People

            • Assignee:
              Navis
              Reporter:
              Navis
            • Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:

                Development