Uploaded image for project: 'Tajo'
  1. Tajo
  2. TAJO-593

outer groupby and groupby in derived table causes only one shuffle output number

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.8.0
    • Component/s: distributed query plan
    • Labels:
      None

      Description

      See the following query case:

      select count(*) from (select l_orderkey, l_partkey, count(*) from lineitem group by l_orderkey, l_partkey) t1;
      

      In this case, SubQuery::calculateShuffleOutputNum() are used two times for choosing the number of shuffle outputs. At that time, SubQuery::calculateShuffleOutputNum() method finds GroupByNode to know the number of grouping keys. Here is one bug. SubQuery::calculateShuffleOutputNum() always the topmost GroupByNode. In most cases, it work well. But, outer groupby and groupby in derived table can cause the problem. In this case, we must use the most bottom groupby node. Actually, it is always the correct way.

      This patch fixes SubQuery::calculateShuffleOutputNum() to use the most bottom groupby node.

      1. TAJO-593.patch
        2 kB
        Hyunsik Choi

        Activity

        Hide
        hyunsik Hyunsik Choi added a comment -

        Created a review request against branch master in reviewboard
        https://reviews.apache.org/r/17905/

        Show
        hyunsik Hyunsik Choi added a comment - Created a review request against branch master in reviewboard https://reviews.apache.org/r/17905/
        Hide
        tajoqa Tajo QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12628001/TAJO-593.patch
        against master revision 4179a7c.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The applied patch does not increase the total number of javadoc warnings.

        +1 checkstyle. The patch generated 0 code style errors.

        -1 findbugs. The patch appears to introduce 192 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in tajo-core/tajo-core-backend.

        Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/116//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/116//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core-backend.html
        Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/116//console

        This message is automatically generated.

        Show
        tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12628001/TAJO-593.patch against master revision 4179a7c. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The applied patch does not increase the total number of javadoc warnings. +1 checkstyle. The patch generated 0 code style errors. -1 findbugs. The patch appears to introduce 192 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in tajo-core/tajo-core-backend. Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/116//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/116//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core-backend.html Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/116//console This message is automatically generated.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Tajo-master-build #61 (See https://builds.apache.org/job/Tajo-master-build/61/)
        TAJO-593: outer groupby and groupby in derived table causes only one shuffle output number. (hyunsik: https://git-wip-us.apache.org/repos/asf?p=incubator-tajo.git&a=commit&h=8e859fb103b753ba6c9379bc7625164341c84b25)

        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/SubQuery.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/PlannerUtil.java
        • CHANGES.txt
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Tajo-master-build #61 (See https://builds.apache.org/job/Tajo-master-build/61/ ) TAJO-593 : outer groupby and groupby in derived table causes only one shuffle output number. (hyunsik: https://git-wip-us.apache.org/repos/asf?p=incubator-tajo.git&a=commit&h=8e859fb103b753ba6c9379bc7625164341c84b25 ) tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/SubQuery.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/PlannerUtil.java CHANGES.txt
        Hide
        hyunsik Hyunsik Choi added a comment -

        This issue got +1 on RB. committed.

        Show
        hyunsik Hyunsik Choi added a comment - This issue got +1 on RB. committed.

          People

          • Assignee:
            hyunsik Hyunsik Choi
            Reporter:
            hyunsik Hyunsik Choi
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development