Uploaded image for project: 'Tajo'
  1. Tajo
  2. TAJO-926

Join condition including column references of a row-preserving table in left outer join causes incorrect result

    Details

      Description

      This patch fixes two bugs.

      One is wrong projection push down (PPD). See the example, reproducing the bug:

      select
        r_name,
        r_regionkey,
        n_name,
        n_regionkey
      from
        region left outer join nation on n_regionkey = r_regionkey and r_name in ('AMERICA', 'ASIA')
      order by r_name;
      

      The above query includes one left outer join (LOJ) and one join filter. Since this join filter R_NAME in ('AMERICA', 'ASIA') includes column references corresponding to the row preserved table region, the join filter is placed on the LOJ operator. It only results in the sub expression push down of RowConstantEval and replaces right expression of IN predicate by FieldEval. But, we assume that the RHS of InEval is always RowConstantEval. This is the main clause of this bug.

      2014-07-09 16:39:37,527 ERROR: org.apache.tajo.worker.Task (run(395)) - org.apache.tajo.engine.eval.FieldEval cannot be cast to org.apache.tajo.engine.eval.RowConstantEval
      java.lang.ClassCastException: org.apache.tajo.engine.eval.FieldEval cannot be cast to org.apache.tajo.engine.eval.RowConstantEval
      	at org.apache.tajo.engine.eval.InEval.eval(InEval.java:62)
      	at org.apache.tajo.engine.eval.BinaryEval.eval(BinaryEval.java:104)
      	at org.apache.tajo.engine.planner.physical.NLLeftOuterJoinExec.next(NLLeftOuterJoinExec.java:109)
      	at org.apache.tajo.engine.planner.physical.ExternalSortExec.sortAndStoreAllChunks(ExternalSortExec.java:201)
      	at org.apache.tajo.engine.planner.physical.ExternalSortExec.next(ExternalSortExec.java:278)
      	at org.apache.tajo.engine.planner.physical.RangeShuffleFileWriteExec.next(RangeShuffleFileWriteExec.java:99)
      	at org.apache.tajo.worker.Task.run(Task.java:388)
      	at org.apache.tajo.worker.TaskRunner$1.run(TaskRunner.java:406)
      	at java.lang.Thread.run(Thread.java:744)
      2014-07-09 16:39:37,528 INFO: org.apache.tajo.worker.TaskAttemptContext (setState(115)) - Query status of ta_1404891573341_0004_000003_000000_02 is changed to TA_FAILED
      2014-07-09 16:39:37,529 INFO: org.apache.tajo.worker.Task (run(452)) - Worker's task counter - total:3, succeeded: 0, killed: 3, failed: 3
      

      The second bug is that HashLeftOuterJoin results in wrong result when it has join filter corresponding to row preserved table like the above example query.
      In order to fix this bug, we have to skip the right iterator of hash table when if the joined tuple is filtered.

      Expected:

      r_name,r_regionkey,n_name,n_regionkey
      -------------------------------
      AFRICA,0,null,null
      AMERICA,1,ARGENTINA,1
      AMERICA,1,BRAZIL,1
      AMERICA,1,CANADA,1
      AMERICA,1,PERU,1
      AMERICA,1,UNITED STATES,1
      ASIA,2,INDIA,2
      ASIA,2,INDONESIA,2
      ASIA,2,JAPAN,2
      ASIA,2,CHINA,2
      ASIA,2,VIETNAM,2
      EUROPE,3,null,null
      MIDDLE EAST,4,null,null
      

      Actual result:

      r_name,r_regionkey,n_name,n_regionkey
      -------------------------------
      AFRICA,0,null,null
      AFRICA,0,null,null
      AFRICA,0,null,null
      AFRICA,0,null,null
      AFRICA,0,null,null
      AMERICA,1,ARGENTINA,1
      AMERICA,1,BRAZIL,1
      AMERICA,1,CANADA,1
      AMERICA,1,PERU,1
      AMERICA,1,UNITED STATES,1
      ASIA,2,INDIA,2
      ASIA,2,INDONESIA,2
      ASIA,2,JAPAN,2
      ASIA,2,CHINA,2
      ASIA,2,VIETNAM,2
      EUROPE,3,null,null
      EUROPE,3,null,null
      EUROPE,3,null,null
      EUROPE,3,null,null
      EUROPE,3,null,null
      MIDDLE EAST,4,null,null
      MIDDLE EAST,4,null,null
      MIDDLE EAST,4,null,null
      MIDDLE EAST,4,null,null
      MIDDLE EAST,4,null,null
      

        Activity

        Hide
        githubbot ASF GitHub Bot added a comment -

        GitHub user hyunsik opened a pull request:

        https://github.com/apache/tajo/pull/62

        TAJO-926: Join condition including column references of a row-preserving table in left outer join causes incorrect result.

        See the jira issue.
        https://issues.apache.org/jira/browse/TAJO-926

        This patch fixes the bug and adds the unit test which produces the bug.

        You can merge this pull request into a Git repository by running:

        $ git pull https://github.com/hyunsik/tajo TAJO-926

        Alternatively you can review and apply these changes as the patch at:

        https://github.com/apache/tajo/pull/62.patch

        To close this pull request, make a commit to your master/trunk branch
        with (at least) the following in the commit message:

        This closes #62


        commit 25ad6d84de992c124651c7e8d3fda4d5ffc6ac8c
        Author: Hyunsik Choi <hyunsik@apache.org>
        Date: 2014-07-09T06:26:46Z

        Fixed bugs of hash outer join operator and wrong PPD against in predicate.

        commit 737896d3baaae217d85df152bed89a4bb37bcbd9
        Author: Hyunsik Choi <hyunsik@apache.org>
        Date: 2014-07-09T06:27:19Z

        Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into HASHOUTER_INPRED_BUG

        commit ab8c2ddb8ab26f3c00bc0002e082a611565a48e2
        Author: Hyunsik Choi <hyunsik@apache.org>
        Date: 2014-07-09T08:14:44Z

        TAJO-926: Join condition including column references of a row-preserving table in left outer join causes incorrect result. (hyunsik)


        Show
        githubbot ASF GitHub Bot added a comment - GitHub user hyunsik opened a pull request: https://github.com/apache/tajo/pull/62 TAJO-926 : Join condition including column references of a row-preserving table in left outer join causes incorrect result. See the jira issue. https://issues.apache.org/jira/browse/TAJO-926 This patch fixes the bug and adds the unit test which produces the bug. You can merge this pull request into a Git repository by running: $ git pull https://github.com/hyunsik/tajo TAJO-926 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/tajo/pull/62.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #62 commit 25ad6d84de992c124651c7e8d3fda4d5ffc6ac8c Author: Hyunsik Choi <hyunsik@apache.org> Date: 2014-07-09T06:26:46Z Fixed bugs of hash outer join operator and wrong PPD against in predicate. commit 737896d3baaae217d85df152bed89a4bb37bcbd9 Author: Hyunsik Choi <hyunsik@apache.org> Date: 2014-07-09T06:27:19Z Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into HASHOUTER_INPRED_BUG commit ab8c2ddb8ab26f3c00bc0002e082a611565a48e2 Author: Hyunsik Choi <hyunsik@apache.org> Date: 2014-07-09T08:14:44Z TAJO-926 : Join condition including column references of a row-preserving table in left outer join causes incorrect result. (hyunsik)
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user hyunsik commented on the pull request:

        https://github.com/apache/tajo/pull/62#issuecomment-48692264

        This is not unit test failure. Due to unknown network problem, it seems to be failed. Please review this.

        Show
        githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on the pull request: https://github.com/apache/tajo/pull/62#issuecomment-48692264 This is not unit test failure. Due to unknown network problem, it seems to be failed. Please review this.
        Hide
        blrunner Jaehwa Jung added a comment -

        +1

        Thank you for your contribution.
        It looks good overall, and 'mvn clean install' finished successfully.
        Push it.

        Show
        blrunner Jaehwa Jung added a comment - +1 Thank you for your contribution. It looks good overall, and 'mvn clean install' finished successfully. Push it.
        Hide
        hyunsik Hyunsik Choi added a comment -

        Thank you for the quick review. I'll commit it soon.

        Show
        hyunsik Hyunsik Choi added a comment - Thank you for the quick review. I'll commit it soon.
        Hide
        hyunsik Hyunsik Choi added a comment -

        committed it to master branch.

        Show
        hyunsik Hyunsik Choi added a comment - committed it to master branch.
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user asfgit closed the pull request at:

        https://github.com/apache/tajo/pull/62

        Show
        githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/tajo/pull/62
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Tajo-master-build #290 (See https://builds.apache.org/job/Tajo-master-build/290/)
        TAJO-926: Join condition including column references of a row-preserving table in left outer join causes incorrect result. (hyunsik: rev c87fe2e8ca135778d3f9c91606bc41c4eb6da5a8)

        • tajo-core/src/test/resources/queries/TestJoinQuery/testJoinFilterOfRowPreservedTable1.sql
        • tajo-core/src/test/resources/results/TestJoinQuery/testJoinFilterOfRowPreservedTable1.result
        • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/HashLeftOuterJoinExec.java
        • tajo-core/src/test/java/org/apache/tajo/engine/query/TestJoinQuery.java
        • CHANGES
        • tajo-core/src/main/java/org/apache/tajo/engine/planner/rewrite/ProjectionPushDownRule.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Tajo-master-build #290 (See https://builds.apache.org/job/Tajo-master-build/290/ ) TAJO-926 : Join condition including column references of a row-preserving table in left outer join causes incorrect result. (hyunsik: rev c87fe2e8ca135778d3f9c91606bc41c4eb6da5a8) tajo-core/src/test/resources/queries/TestJoinQuery/testJoinFilterOfRowPreservedTable1.sql tajo-core/src/test/resources/results/TestJoinQuery/testJoinFilterOfRowPreservedTable1.result tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/HashLeftOuterJoinExec.java tajo-core/src/test/java/org/apache/tajo/engine/query/TestJoinQuery.java CHANGES tajo-core/src/main/java/org/apache/tajo/engine/planner/rewrite/ProjectionPushDownRule.java

          People

          • Assignee:
            hyunsik Hyunsik Choi
            Reporter:
            hyunsik Hyunsik Choi
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development