Uploaded image for project: 'Tajo'
  1. Tajo
  2. TAJO-583

Broadcast join does not work on partitioned tables

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.8.0
    • Component/s: Planner/Optimizer
    • Labels:
      None

      Description

      Broadcast join does not work on partitioned tables. This is because broadcast join condition just checks ScanNode whose data volume is smaller than some threshold. We should also consider PartitionedTableScan.

      1. TAJO-583_140207_141152.patch
        77 kB
        Hyunsik Choi
      2. TAJO-583.patch
        10 kB
        Hyunsik Choi

        Activity

        Hide
        hyunsik Hyunsik Choi added a comment -

        Created a review request against branch master in reviewboard
        https://reviews.apache.org/r/17728/

        Show
        hyunsik Hyunsik Choi added a comment - Created a review request against branch master in reviewboard https://reviews.apache.org/r/17728/
        Hide
        tajoqa Tajo QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12627042/TAJO-583.patch
        against master revision 5177dcf.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The applied patch does not increase the total number of javadoc warnings.

        +1 checkstyle. The patch generated 0 code style errors.

        -1 findbugs. The patch appears to introduce 191 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in tajo-core/tajo-core-backend.

        Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/106//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/106//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core-backend.html
        Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/106//console

        This message is automatically generated.

        Show
        tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12627042/TAJO-583.patch against master revision 5177dcf. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The applied patch does not increase the total number of javadoc warnings. +1 checkstyle. The patch generated 0 code style errors. -1 findbugs. The patch appears to introduce 191 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in tajo-core/tajo-core-backend. Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/106//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/106//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core-backend.html Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/106//console This message is automatically generated.
        Hide
        hyunsik Hyunsik Choi added a comment -

        Added additional unit tests and fixed found bugs.

        Updated the review request against branch master in reviewboard
        https://reviews.apache.org/r/17728/

        Show
        hyunsik Hyunsik Choi added a comment - Added additional unit tests and fixed found bugs. Updated the review request against branch master in reviewboard https://reviews.apache.org/r/17728/
        Hide
        hyunsik Hyunsik Choi added a comment -

        I've tested all kinds of combinations for joins on partitioned tables. For example, I did the following tests:

        • large partitioned table x small non-partitioned table (broadcasted)
        • few partitions of table (broadcasted) x large non-partitioned partition

        In addition, I tested the case of both are small, leading a two way join on single machine.

        All work well.

        Show
        hyunsik Hyunsik Choi added a comment - I've tested all kinds of combinations for joins on partitioned tables. For example, I did the following tests: large partitioned table x small non-partitioned table (broadcasted) few partitions of table (broadcasted) x large non-partitioned partition In addition, I tested the case of both are small, leading a two way join on single machine. All work well.
        Hide
        hyunsik Hyunsik Choi added a comment -

        This got +1 on RB. committed it to master branch.

        Show
        hyunsik Hyunsik Choi added a comment - This got +1 on RB. committed it to master branch.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Tajo-master-build #59 (See https://builds.apache.org/job/Tajo-master-build/59/)
        TAJO-583: Broadcast join does not work on partitioned tables. (hyunsik: https://git-wip-us.apache.org/repos/asf?p=incubator-tajo.git&a=commit&h=4179a7c9265123bae0a81cacd5eee53a96f98836)

        • tajo-storage/src/main/java/org/apache/tajo/storage/CSVFile.java
        • tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/TpchTestBase.java
        • tajo-core/tajo-core-backend/src/test/resources/results/TestGroupByQuery/testGroupByNested1.result
        • tajo-core/tajo-core-backend/src/test/resources/queries/default/create_partitioned_table_as_select.sql
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/PlanString.java
        • tajo-core/tajo-core-backend/src/test/resources/results/TestJoinOnPartitionedTables/selfJoinOfPartitionedTable.result
        • tajo-core/tajo-core-backend/src/test/resources/queries/TestJoinQuery/testLeftOuterJoin1.sql
        • tajo-core/tajo-core-backend/src/test/resources/queries/TestJoinOnPartitionedTables/insert_into_customer.sql
        • tajo-core/tajo-core-backend/src/test/resources/queries/TestJoinQuery/testLeftOuterJoinWithConstantExpr3.sql
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/global/MasterPlan.java
        • tajo-core/tajo-core-backend/src/test/resources/queries/TestJoinQuery/testLeftOuterJoinWithConstantExpr2.sql
        • tajo-core/tajo-core-backend/src/test/resources/queries/TestJoinOnPartitionedTables/testNoProjectionJoinQual.sql
        • tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/LocalTajoTestingUtility.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/Repartitioner.java
        • tajo-core/tajo-core-backend/src/test/resources/queries/TestJoinOnPartitionedTables/customer_ddl.sql
        • tajo-core/tajo-core-backend/src/test/resources/queries/TestJoinQuery/testLeftOuterJoinWithConstantExpr1.sql
        • tajo-core/tajo-core-backend/src/test/resources/queries/TestJoinOnPartitionedTables/selfJoinOfPartitionedTable.sql
        • tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/planner/physical/TestSortExec.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/rewrite/ProjectionPushDownRule.java
        • tajo-common/src/main/java/org/apache/tajo/util/TUtil.java
        • tajo-core/tajo-core-backend/src/test/resources/results/TestJoinOnPartitionedTables/testNoProjectionJoinQual.result
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/HashLeftOuterJoinExec.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/global/ExecutionBlock.java
        • tajo-core/tajo-core-backend/src/test/resources/queries/TestTablePartitions/case3.sql
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/PlannerUtil.java
        • tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/QueryTestCaseBase.java
        • tajo-core/tajo-core-backend/src/test/resources/queries/TestJoinQuery/testFullOuterJoin1.sql
        • tajo-core/tajo-core-backend/src/test/resources/queries/TestJoinOnPartitionedTables/testPartitionTableJoinSmallTable.sql
        • tajo-core/tajo-core-backend/src/test/resources/results/TestJoinOnPartitionedTables/testPartialFilterPushDown.result
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/PhysicalPlannerImpl.java
        • tajo-core/tajo-core-backend/src/test/resources/results/TestTablePartitions/case3.result
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/SubQuery.java
        • tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/query/TestCTASQuery.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/rewrite/FilterPushDownRule.java
        • tajo-core/tajo-core-backend/src/test/resources/queries/TestJoinQuery/testRightOuterJoin1.sql
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/global/GlobalPlanner.java
        • tajo-core/tajo-core-backend/src/test/resources/queries/TestGroupByQuery/testGroupByNested2.sql
        • tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/TajoTestingCluster.java
        • tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/query/TestTablePartitions.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/PartitionMergeScanExec.java
        • tajo-core/tajo-core-backend/src/test/resources/queries/TestJoinOnPartitionedTables/testPartialFilterPushDown.sql
        • CHANGES.txt
        • tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/query/TestJoinOnPartitionedTables.java
        • tajo-core/tajo-core-backend/src/test/resources/results/TestJoinOnPartitionedTables/testPartitionTableJoinSmallTable.result
        • tajo-core/tajo-core-backend/src/test/resources/results/TestGroupByQuery/testGroupByNested2.result
        • tajo-core/tajo-core-backend/src/test/resources/queries/TestGroupByQuery/testGroupByNested1.sql
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Tajo-master-build #59 (See https://builds.apache.org/job/Tajo-master-build/59/ ) TAJO-583 : Broadcast join does not work on partitioned tables. (hyunsik: https://git-wip-us.apache.org/repos/asf?p=incubator-tajo.git&a=commit&h=4179a7c9265123bae0a81cacd5eee53a96f98836 ) tajo-storage/src/main/java/org/apache/tajo/storage/CSVFile.java tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/TpchTestBase.java tajo-core/tajo-core-backend/src/test/resources/results/TestGroupByQuery/testGroupByNested1.result tajo-core/tajo-core-backend/src/test/resources/queries/default/create_partitioned_table_as_select.sql tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/PlanString.java tajo-core/tajo-core-backend/src/test/resources/results/TestJoinOnPartitionedTables/selfJoinOfPartitionedTable.result tajo-core/tajo-core-backend/src/test/resources/queries/TestJoinQuery/testLeftOuterJoin1.sql tajo-core/tajo-core-backend/src/test/resources/queries/TestJoinOnPartitionedTables/insert_into_customer.sql tajo-core/tajo-core-backend/src/test/resources/queries/TestJoinQuery/testLeftOuterJoinWithConstantExpr3.sql tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/global/MasterPlan.java tajo-core/tajo-core-backend/src/test/resources/queries/TestJoinQuery/testLeftOuterJoinWithConstantExpr2.sql tajo-core/tajo-core-backend/src/test/resources/queries/TestJoinOnPartitionedTables/testNoProjectionJoinQual.sql tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/LocalTajoTestingUtility.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/Repartitioner.java tajo-core/tajo-core-backend/src/test/resources/queries/TestJoinOnPartitionedTables/customer_ddl.sql tajo-core/tajo-core-backend/src/test/resources/queries/TestJoinQuery/testLeftOuterJoinWithConstantExpr1.sql tajo-core/tajo-core-backend/src/test/resources/queries/TestJoinOnPartitionedTables/selfJoinOfPartitionedTable.sql tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/planner/physical/TestSortExec.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/rewrite/ProjectionPushDownRule.java tajo-common/src/main/java/org/apache/tajo/util/TUtil.java tajo-core/tajo-core-backend/src/test/resources/results/TestJoinOnPartitionedTables/testNoProjectionJoinQual.result tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/HashLeftOuterJoinExec.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/global/ExecutionBlock.java tajo-core/tajo-core-backend/src/test/resources/queries/TestTablePartitions/case3.sql tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/PlannerUtil.java tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/QueryTestCaseBase.java tajo-core/tajo-core-backend/src/test/resources/queries/TestJoinQuery/testFullOuterJoin1.sql tajo-core/tajo-core-backend/src/test/resources/queries/TestJoinOnPartitionedTables/testPartitionTableJoinSmallTable.sql tajo-core/tajo-core-backend/src/test/resources/results/TestJoinOnPartitionedTables/testPartialFilterPushDown.result tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/PhysicalPlannerImpl.java tajo-core/tajo-core-backend/src/test/resources/results/TestTablePartitions/case3.result tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/SubQuery.java tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/query/TestCTASQuery.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/rewrite/FilterPushDownRule.java tajo-core/tajo-core-backend/src/test/resources/queries/TestJoinQuery/testRightOuterJoin1.sql tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/global/GlobalPlanner.java tajo-core/tajo-core-backend/src/test/resources/queries/TestGroupByQuery/testGroupByNested2.sql tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/TajoTestingCluster.java tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/query/TestTablePartitions.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/PartitionMergeScanExec.java tajo-core/tajo-core-backend/src/test/resources/queries/TestJoinOnPartitionedTables/testPartialFilterPushDown.sql CHANGES.txt tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/query/TestJoinOnPartitionedTables.java tajo-core/tajo-core-backend/src/test/resources/results/TestJoinOnPartitionedTables/testPartitionTableJoinSmallTable.result tajo-core/tajo-core-backend/src/test/resources/results/TestGroupByQuery/testGroupByNested2.result tajo-core/tajo-core-backend/src/test/resources/queries/TestGroupByQuery/testGroupByNested1.sql

          People

          • Assignee:
            hyunsik Hyunsik Choi
            Reporter:
            hyunsik Hyunsik Choi
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development