Hive
  1. Hive
  2. HIVE-3617

Predicates pushed down to hbase is not handled properly when constant part is shown first

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.13.0
    • Component/s: HBase Handler
    • Labels:
      None

      Description

      Test result could not show the difference because predicates pushed down are not removed currently(HIVE-2897). So I added log message(scan.toMap()) and checked the output.

      with query
      select * from hbase_ppd_keyrange where key > 8 and key < 21;
      timeRange=[0, 9223372036854775807], batch=-1, startRow=\x00\x00\x00\x08\x00, stopRow=\x00\x00\x00\x15, ...

      but with query
      select * from hbase_ppd_keyrange where 8 < key and key < 21;
      timeRange=[0, 9223372036854775807], batch=-1, startRow=, stopRow=\x00\x00\x00\x15, ...

        Issue Links

          Activity

          Hide
          Navis added a comment -

          Committed to trunk. Thanks, Ashutosh.

          Show
          Navis added a comment - Committed to trunk. Thanks, Ashutosh.
          Hide
          Ashutosh Chauhan added a comment -

          Ya, you are correct. Also, some of constant folding code in here won't be needed after HIVE-5771 perhaps we can simplify that whenever thats gets checked in.
          +1 lets get this one in.

          Show
          Ashutosh Chauhan added a comment - Ya, you are correct. Also, some of constant folding code in here won't be needed after HIVE-5771 perhaps we can simplify that whenever thats gets checked in. +1 lets get this one in.
          Hide
          Navis added a comment -

          I also confused with that. negate() had the semantic as you described (> for <=, etc), which is now removed.

          In a word, "3<a" is "a>3", not "a>=3"

          Show
          Navis added a comment - I also confused with that. negate() had the semantic as you described (> for <=, etc), which is now removed. In a word, "3<a" is "a>3", not "a>=3"
          Hide
          Ashutosh Chauhan added a comment -

          Patch looks good. I may be confused but won't flip for >= should be <, instead of <=. Similarly, for <, flip should be >= and so on...

          Show
          Ashutosh Chauhan added a comment - Patch looks good. I may be confused but won't flip for >= should be <, instead of <=. Similarly, for <, flip should be >= and so on...
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12623853/HIVE-3617.3.patch.txt

          ERROR: -1 due to 1 failed/errored test(s), 4943 tests executed
          Failed tests:

          org.apache.hadoop.hive.ql.metadata.TestHiveRemote.testTable
          

          Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/960/testReport
          Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/960/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 1 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12623853

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12623853/HIVE-3617.3.patch.txt ERROR: -1 due to 1 failed/errored test(s), 4943 tests executed Failed tests: org.apache.hadoop.hive.ql.metadata.TestHiveRemote.testTable Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/960/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/960/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed This message is automatically generated. ATTACHMENT ID: 12623853
          Hide
          Ashutosh Chauhan added a comment -

          I don't see where you are using newly introduced negate() in this patch? Can that be removed from the patch? Also, patch needs to be rebased on trunk.

          Show
          Ashutosh Chauhan added a comment - I don't see where you are using newly introduced negate() in this patch? Can that be removed from the patch? Also, patch needs to be rebased on trunk.
          Hide
          Phabricator added a comment -

          navis updated the revision "HIVE-3617 [jira] Predicates pushed down to hbase is not handled properly when constant part is shown first".
          Reviewers: JIRA

          add filp method to genericUDF

          REVISION DETAIL
          https://reviews.facebook.net/D6183

          AFFECTED FILES
          ql/src/java/org/apache/hadoop/hive/ql/index/IndexPredicateAnalyzer.java
          ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java
          ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java
          ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBridge.java
          ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPEqual.java
          ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPEqualNS.java
          ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPEqualOrGreaterThan.java
          ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPEqualOrLessThan.java
          ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPGreaterThan.java
          ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPLessThan.java
          ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPNotEqual.java
          ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPNotNull.java
          ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPNull.java

          To: JIRA, navis

          Show
          Phabricator added a comment - navis updated the revision " HIVE-3617 [jira] Predicates pushed down to hbase is not handled properly when constant part is shown first". Reviewers: JIRA add filp method to genericUDF REVISION DETAIL https://reviews.facebook.net/D6183 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/index/IndexPredicateAnalyzer.java ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBridge.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPEqual.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPEqualNS.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPEqualOrGreaterThan.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPEqualOrLessThan.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPGreaterThan.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPLessThan.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPNotEqual.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPNotNull.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPNull.java To: JIRA, navis
          Hide
          Phabricator added a comment -

          navis requested code review of "HIVE-3617 [jira] Predicates pushed down to hbase is not handled properly when constant part is shown first".
          Reviewers: JIRA

          DPAL-1922 Predicates pushed down to hbase is not handled properly when constant part is shown first

          Test result could not show the difference because predicates pushed down are not removed currently(HIVE-2897). So I added log message(scan.toMap()) and checked the output.

          with query
          select * from hbase_ppd_keyrange where key > 8 and key < 21;
          timeRange=[0, 9223372036854775807], batch=-1, startRow=\x00\x00\x00\x08\x00, stopRow=\x00\x00\x00\x15, ...

          but with query
          select * from hbase_ppd_keyrange where 8 < key and key < 21;
          timeRange=[0, 9223372036854775807], batch=-1, startRow=, stopRow=\x00\x00\x00\x15, ...

          TEST PLAN
          EMPTY

          REVISION DETAIL
          https://reviews.facebook.net/D6183

          AFFECTED FILES
          hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java
          ql/src/java/org/apache/hadoop/hive/ql/index/IndexPredicateAnalyzer.java
          ql/src/java/org/apache/hadoop/hive/ql/index/IndexSearchCondition.java
          ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java

          MANAGE HERALD DIFFERENTIAL RULES
          https://reviews.facebook.net/herald/view/differential/

          WHY DID I GET THIS EMAIL?
          https://reviews.facebook.net/herald/transcript/14763/

          To: JIRA, navis

          Show
          Phabricator added a comment - navis requested code review of " HIVE-3617 [jira] Predicates pushed down to hbase is not handled properly when constant part is shown first". Reviewers: JIRA DPAL-1922 Predicates pushed down to hbase is not handled properly when constant part is shown first Test result could not show the difference because predicates pushed down are not removed currently( HIVE-2897 ). So I added log message(scan.toMap()) and checked the output. with query select * from hbase_ppd_keyrange where key > 8 and key < 21; timeRange= [0, 9223372036854775807] , batch=-1, startRow=\x00\x00\x00\x08\x00, stopRow=\x00\x00\x00\x15, ... but with query select * from hbase_ppd_keyrange where 8 < key and key < 21; timeRange= [0, 9223372036854775807] , batch=-1, startRow=, stopRow=\x00\x00\x00\x15, ... TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D6183 AFFECTED FILES hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java ql/src/java/org/apache/hadoop/hive/ql/index/IndexPredicateAnalyzer.java ql/src/java/org/apache/hadoop/hive/ql/index/IndexSearchCondition.java ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/14763/ To: JIRA, navis

            People

            • Assignee:
              Navis
              Reporter:
              Navis
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development