Hive
  1. Hive
  2. HIVE-5552

Merging of QBJoinTrees doesn't handle filter pushdowns correctly

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.13.0
    • Component/s: Query Processor
    • Labels:
      None

      Description

      The following query fails:
      (this based on the schema from auto_join_filters.q)

      explain
      SELECT sum(hash(a.key,a.value,b.key,b.value)) 
      FROM myinput1 a LEFT OUTER JOIN myinput1 b on a.value = b.value  RIGHT OUTER JOIN myinput1 c 
           ON 
              b.value = c.value AND 
              a.key > 40
      

      Whereas this query succeeds

      explain
      SELECT sum(hash(a.key,a.value,b.key,b.value)) 
      FROM myinput1 a LEFT OUTER JOIN myinput1 b RIGHT OUTER JOIN myinput1 c 
           ON  a.value = b.value and
              b.value = c.value AND 
              a.key > 40
      

      Pushing the first condition to the first join, triggers a merge of the 2 QBJoinTrees. During merge all the right side filters identified for pushing are assumed to refer to the merging table (b in this case). But the pushable filters can refer to any left table.

      1. HIVE-5552.1.patch
        14 kB
        Harish Butani
      2. HIVE-5552.2.patch
        14 kB
        Harish Butani

        Issue Links

          Activity

          Hide
          Ashutosh Chauhan added a comment -

          Harish, left some comments on RB. Also, once you update the patch, also upload it here so that HiveQA gets to run on it.

          Show
          Ashutosh Chauhan added a comment - Harish, left some comments on RB. Also, once you update the patch, also upload it here so that HiveQA gets to run on it.
          Hide
          Ashutosh Chauhan added a comment -

          +1

          Show
          Ashutosh Chauhan added a comment - +1
          Hide
          Hive QA added a comment -

          Overall: +1 all checks pass

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12610330/HIVE-5552.2.patch

          SUCCESS: +1 4482 tests passed

          Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1236/testReport
          Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1236/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          

          This message is automatically generated.

          Show
          Hive QA added a comment - Overall : +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12610330/HIVE-5552.2.patch SUCCESS: +1 4482 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1236/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1236/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated.
          Hide
          Ashutosh Chauhan added a comment -

          Committed to trunk. Thanks, Harish!

          Show
          Ashutosh Chauhan added a comment - Committed to trunk. Thanks, Harish!
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hive-trunk-h0.21 #2422 (See https://builds.apache.org/job/Hive-trunk-h0.21/2422/)
          HIVE-5552 : Merging of QBJoinTrees doesnt handle filter pushdowns correctly (Harish Butani via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1535858)

          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseUtils.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/QBJoinTree.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
          • /hive/trunk/ql/src/test/queries/clientpositive/join_merging.q
          • /hive/trunk/ql/src/test/results/clientpositive/join_merging.q.out
          Show
          Hudson added a comment - FAILURE: Integrated in Hive-trunk-h0.21 #2422 (See https://builds.apache.org/job/Hive-trunk-h0.21/2422/ ) HIVE-5552 : Merging of QBJoinTrees doesnt handle filter pushdowns correctly (Harish Butani via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1535858 ) /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseUtils.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/QBJoinTree.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java /hive/trunk/ql/src/test/queries/clientpositive/join_merging.q /hive/trunk/ql/src/test/results/clientpositive/join_merging.q.out
          Hide
          Hudson added a comment -

          ABORTED: Integrated in Hive-trunk-hadoop2 #522 (See https://builds.apache.org/job/Hive-trunk-hadoop2/522/)
          HIVE-5552 : Merging of QBJoinTrees doesnt handle filter pushdowns correctly (Harish Butani via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1535858)

          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseUtils.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/QBJoinTree.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
          • /hive/trunk/ql/src/test/queries/clientpositive/join_merging.q
          • /hive/trunk/ql/src/test/results/clientpositive/join_merging.q.out
          Show
          Hudson added a comment - ABORTED: Integrated in Hive-trunk-hadoop2 #522 (See https://builds.apache.org/job/Hive-trunk-hadoop2/522/ ) HIVE-5552 : Merging of QBJoinTrees doesnt handle filter pushdowns correctly (Harish Butani via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1535858 ) /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseUtils.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/QBJoinTree.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java /hive/trunk/ql/src/test/queries/clientpositive/join_merging.q /hive/trunk/ql/src/test/results/clientpositive/join_merging.q.out
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hive-trunk-hadoop2-ptest #152 (See https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/152/)
          HIVE-5552 : Merging of QBJoinTrees doesnt handle filter pushdowns correctly (Harish Butani via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1535858)

          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseUtils.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/QBJoinTree.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
          • /hive/trunk/ql/src/test/queries/clientpositive/join_merging.q
          • /hive/trunk/ql/src/test/results/clientpositive/join_merging.q.out
          Show
          Hudson added a comment - FAILURE: Integrated in Hive-trunk-hadoop2-ptest #152 (See https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/152/ ) HIVE-5552 : Merging of QBJoinTrees doesnt handle filter pushdowns correctly (Harish Butani via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1535858 ) /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseUtils.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/QBJoinTree.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java /hive/trunk/ql/src/test/queries/clientpositive/join_merging.q /hive/trunk/ql/src/test/results/clientpositive/join_merging.q.out
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Hive-trunk-hadoop1-ptest #215 (See https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/215/)
          HIVE-5552 : Merging of QBJoinTrees doesnt handle filter pushdowns correctly (Harish Butani via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1535858)

          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseUtils.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/QBJoinTree.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
          • /hive/trunk/ql/src/test/queries/clientpositive/join_merging.q
          • /hive/trunk/ql/src/test/results/clientpositive/join_merging.q.out
          Show
          Hudson added a comment - SUCCESS: Integrated in Hive-trunk-hadoop1-ptest #215 (See https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/215/ ) HIVE-5552 : Merging of QBJoinTrees doesnt handle filter pushdowns correctly (Harish Butani via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1535858 ) /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseUtils.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/QBJoinTree.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java /hive/trunk/ql/src/test/queries/clientpositive/join_merging.q /hive/trunk/ql/src/test/results/clientpositive/join_merging.q.out

            People

            • Assignee:
              Harish Butani
              Reporter:
              Harish Butani
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development