Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.11.0
    • Component/s: None
    • Labels:
      None

      Description

      As discussed in TAJO-1517,
      INTERSECT ALL needs new physical operator,
      which returns the exact number of overlapping rows between results of two query blocks.
      For example, when query1 returns

      1, 3, 3, 3
      

      while query2 returns

      3, 3
      

      query1 INTERSECT ALL query2 should return

      3, 3
      

      INTERSECT can be implemented by selecting distinct result of above physical operator.

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Tajo-master-build #757 (See https://builds.apache.org/job/Tajo-master-build/757/)
          TAJO-1525: Implement INTERSECT [ALL] physical operator. (Contributed Keuntae Park, committed by hyunsik) (hyunsik: rev 7481050e3d75eb860590f3237aca210fdd98190a)

          • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/SetTupleComparator.java
          • CHANGES
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/SortIntersectExec.java
          • tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestSortIntersectExec.java
            TAJO-1525: Implement INTERSECT [ALL] physical operator. (missed issue close) (hyunsik: rev c50a5dadff90fa90709abbce59856e834baa4867)
          • CHANGES
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Tajo-master-build #757 (See https://builds.apache.org/job/Tajo-master-build/757/ ) TAJO-1525 : Implement INTERSECT [ALL] physical operator. (Contributed Keuntae Park, committed by hyunsik) (hyunsik: rev 7481050e3d75eb860590f3237aca210fdd98190a) tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/SetTupleComparator.java CHANGES tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/SortIntersectExec.java tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestSortIntersectExec.java TAJO-1525 : Implement INTERSECT [ALL] physical operator. (missed issue close) (hyunsik: rev c50a5dadff90fa90709abbce59856e834baa4867) CHANGES
          Hide
          hudson Hudson added a comment -

          ABORTED: Integrated in Tajo-master-CODEGEN-build #397 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/397/)
          TAJO-1525: Implement INTERSECT [ALL] physical operator. (Contributed Keuntae Park, committed by hyunsik) (hyunsik: rev 7481050e3d75eb860590f3237aca210fdd98190a)

          • CHANGES
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/SortIntersectExec.java
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/SetTupleComparator.java
          • tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestSortIntersectExec.java
            TAJO-1525: Implement INTERSECT [ALL] physical operator. (missed issue close) (hyunsik: rev c50a5dadff90fa90709abbce59856e834baa4867)
          • CHANGES
          Show
          hudson Hudson added a comment - ABORTED: Integrated in Tajo-master-CODEGEN-build #397 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/397/ ) TAJO-1525 : Implement INTERSECT [ALL] physical operator. (Contributed Keuntae Park, committed by hyunsik) (hyunsik: rev 7481050e3d75eb860590f3237aca210fdd98190a) CHANGES tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/SortIntersectExec.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/SetTupleComparator.java tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestSortIntersectExec.java TAJO-1525 : Implement INTERSECT [ALL] physical operator. (missed issue close) (hyunsik: rev c50a5dadff90fa90709abbce59856e834baa4867) CHANGES
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/tajo/pull/513

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/tajo/pull/513
          Hide
          hyunsik Hyunsik Choi added a comment - - edited

          committed the patch to master branch. Thank you for your contribution.

          Show
          hyunsik Hyunsik Choi added a comment - - edited committed the patch to master branch. Thank you for your contribution.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on the pull request:

          https://github.com/apache/tajo/pull/513#issuecomment-122865647

          +1
          The physical operator works well, and the patch looks good to me. I'll finish the logical plan and global planner part in the following issue. Thank you for your work.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on the pull request: https://github.com/apache/tajo/pull/513#issuecomment-122865647 +1 The physical operator works well, and the patch looks good to me. I'll finish the logical plan and global planner part in the following issue. Thank you for your work.
          Hide
          sirpkt Keuntae Park added a comment -

          At first, I thought that new physical operator is only for INTERSECT ALL and
          INTERSECT would be handled by INTERSECT ALL + distinct.
          But, during implementation, I found that single physical operator for INTERSECT and INTERSECT ALL is much better approach
          because it results in much simpler plan for INTERSECT.
          So, I'll update patch to implement single physical operator to support both INTERSECT and INTERSECT ALL.

          Show
          sirpkt Keuntae Park added a comment - At first, I thought that new physical operator is only for INTERSECT ALL and INTERSECT would be handled by INTERSECT ALL + distinct. But, during implementation, I found that single physical operator for INTERSECT and INTERSECT ALL is much better approach because it results in much simpler plan for INTERSECT. So, I'll update patch to implement single physical operator to support both INTERSECT and INTERSECT ALL.
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user sirpkt opened a pull request:

          https://github.com/apache/tajo/pull/513

          TAJO-1525: Implement INTERSECT ALL physical operator

          This patch implements sort based INTERSECT ALL physical operator.

          INTERSECT ALL operator has two input physical operators that should return rows sorted by all the schema columns.
          Then, it iterates results of both children operators and returns the common rows from both.

          • SetTupleComparator is implemented to compare the whole columns of given two tuples.
          • Test code is also included, which tests INTERSECT ALL exec by replacing inner join with it. It is somewhat tricky, however, useful way when Logical Planner and Global Planner are not ready for INTERSECT [ALL].

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/sirpkt/tajo TAJO-1525

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/tajo/pull/513.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #513


          commit fdba6a66a8bb71fee14468929251cb87c624ff27
          Author: Keuntae Park <sirpkt@apache.org>
          Date: 2015-04-06T06:41:38Z

          Implement SortIntersetAllExec, sort based INTERSECT ALL physical operator


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user sirpkt opened a pull request: https://github.com/apache/tajo/pull/513 TAJO-1525 : Implement INTERSECT ALL physical operator This patch implements sort based INTERSECT ALL physical operator. INTERSECT ALL operator has two input physical operators that should return rows sorted by all the schema columns. Then, it iterates results of both children operators and returns the common rows from both. SetTupleComparator is implemented to compare the whole columns of given two tuples. Test code is also included, which tests INTERSECT ALL exec by replacing inner join with it. It is somewhat tricky, however, useful way when Logical Planner and Global Planner are not ready for INTERSECT [ALL] . You can merge this pull request into a Git repository by running: $ git pull https://github.com/sirpkt/tajo TAJO-1525 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/tajo/pull/513.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #513 commit fdba6a66a8bb71fee14468929251cb87c624ff27 Author: Keuntae Park <sirpkt@apache.org> Date: 2015-04-06T06:41:38Z Implement SortIntersetAllExec, sort based INTERSECT ALL physical operator

            People

            • Assignee:
              sirpkt Keuntae Park
              Reporter:
              sirpkt Keuntae Park
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development