Uploaded image for project: 'Tajo'
  1. Tajo
  2. TAJO-1049

Remove the parallel degree limit up to the maximum cluster capacity

    Details

      Description

      The maximum parallel degree had been limited by the number of cluster node capacity. This was because the old hash shuffle approach created too many small files depending on the number of parallel degree of a stage.

      But, TAJO-992 solved this problem. We don't need to use this limit anymore. This patch removes the limit.

      I think that this patch requires cluster test and it is hard to verify this patch in unit tests.

      1. TAJO-1049_2.patch
        2 kB
        Hyunsik Choi
      2. TAJO-1049_3.patch
        2 kB
        Hyunsik Choi
      3. TAJO-1049.patch
        1 kB
        Hyunsik Choi

        Activity

        Hide
        githubbot ASF GitHub Bot added a comment -

        GitHub user hyunsik opened a pull request:

        https://github.com/apache/tajo/pull/147

        TAJO-1049: Remove the parallel degree limit up to the maximum cluster ca...

        ...pacity.

        You can merge this pull request into a Git repository by running:

        $ git pull https://github.com/hyunsik/tajo TAJO-1049

        Alternatively you can review and apply these changes as the patch at:

        https://github.com/apache/tajo/pull/147.patch

        To close this pull request, make a commit to your master/trunk branch
        with (at least) the following in the commit message:

        This closes #147


        commit bdeedac47697b072f8d32e5673a4651e611bf6c7
        Author: Hyunsik Choi <hyunsik@apache.org>
        Date: 2014-09-17T15:37:01Z

        TAJO-1049: Remove the parallel degree limit up to the maximum cluster capacity.


        Show
        githubbot ASF GitHub Bot added a comment - GitHub user hyunsik opened a pull request: https://github.com/apache/tajo/pull/147 TAJO-1049 : Remove the parallel degree limit up to the maximum cluster ca... ...pacity. You can merge this pull request into a Git repository by running: $ git pull https://github.com/hyunsik/tajo TAJO-1049 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/tajo/pull/147.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #147 commit bdeedac47697b072f8d32e5673a4651e611bf6c7 Author: Hyunsik Choi <hyunsik@apache.org> Date: 2014-09-17T15:37:01Z TAJO-1049 : Remove the parallel degree limit up to the maximum cluster capacity.
        Hide
        tajoqa Tajo QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12669424/TAJO-1049.patch
        against master revision 1a8a67c.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The applied patch does not increase the total number of javadoc warnings.

        +1 checkstyle. The patch generated 0 code style errors.

        -1 findbugs. The patch appears to introduce 209 new Findbugs (version 2.0.3) warnings.

        -1 release audit. The applied patch generated 104 release audit warnings.

        +1 core tests. The patch passed unit tests in tajo-core.

        Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/491//testReport/
        Release audit warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/491//artifact/incubator-tajo/patchprocess/patchReleaseAuditProblems.txt
        Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/491//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core.html
        Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/491//console

        This message is automatically generated.

        Show
        tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12669424/TAJO-1049.patch against master revision 1a8a67c. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The applied patch does not increase the total number of javadoc warnings. +1 checkstyle. The patch generated 0 code style errors. -1 findbugs. The patch appears to introduce 209 new Findbugs (version 2.0.3) warnings. -1 release audit. The applied patch generated 104 release audit warnings. +1 core tests. The patch passed unit tests in tajo-core. Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/491//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/491//artifact/incubator-tajo/patchprocess/patchReleaseAuditProblems.txt Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/491//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core.html Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/491//console This message is automatically generated.
        Hide
        tajoqa Tajo QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12669428/TAJO-1049_2.patch
        against master revision 1a8a67c.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The applied patch does not increase the total number of javadoc warnings.

        +1 checkstyle. The patch generated 0 code style errors.

        -1 findbugs. The patch appears to introduce 209 new Findbugs (version 2.0.3) warnings.

        -1 release audit. The applied patch generated 103 release audit warnings.

        +1 core tests. The patch passed unit tests in tajo-core.

        Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/492//testReport/
        Release audit warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/492//artifact/incubator-tajo/patchprocess/patchReleaseAuditProblems.txt
        Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/492//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core.html
        Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/492//console

        This message is automatically generated.

        Show
        tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12669428/TAJO-1049_2.patch against master revision 1a8a67c. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The applied patch does not increase the total number of javadoc warnings. +1 checkstyle. The patch generated 0 code style errors. -1 findbugs. The patch appears to introduce 209 new Findbugs (version 2.0.3) warnings. -1 release audit. The applied patch generated 103 release audit warnings. +1 core tests. The patch passed unit tests in tajo-core. Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/492//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/492//artifact/incubator-tajo/patchprocess/patchReleaseAuditProblems.txt Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/492//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core.html Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/492//console This message is automatically generated.
        Hide
        tajoqa Tajo QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12669435/TAJO-1049_3.patch
        against master revision 745131e.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The applied patch does not increase the total number of javadoc warnings.

        +1 checkstyle. The patch generated 0 code style errors.

        -1 findbugs. The patch appears to introduce 210 new Findbugs (version 2.0.3) warnings.

        -1 release audit. The applied patch generated 105 release audit warnings.

        +1 core tests. The patch passed unit tests in tajo-core.

        Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/493//testReport/
        Release audit warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/493//artifact/incubator-tajo/patchprocess/patchReleaseAuditProblems.txt
        Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/493//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core.html
        Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/493//console

        This message is automatically generated.

        Show
        tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12669435/TAJO-1049_3.patch against master revision 745131e. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The applied patch does not increase the total number of javadoc warnings. +1 checkstyle. The patch generated 0 code style errors. -1 findbugs. The patch appears to introduce 210 new Findbugs (version 2.0.3) warnings. -1 release audit. The applied patch generated 105 release audit warnings. +1 core tests. The patch passed unit tests in tajo-core. Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/493//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/493//artifact/incubator-tajo/patchprocess/patchReleaseAuditProblems.txt Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/493//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core.html Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/493//console This message is automatically generated.
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user jinossy commented on the pull request:

        https://github.com/apache/tajo/pull/147#issuecomment-56259080

        +1
        Cluster limitation was temporary solution.

        Show
        githubbot ASF GitHub Bot added a comment - Github user jinossy commented on the pull request: https://github.com/apache/tajo/pull/147#issuecomment-56259080 +1 Cluster limitation was temporary solution.
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user asfgit closed the pull request at:

        https://github.com/apache/tajo/pull/147

        Show
        githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/tajo/pull/147
        Hide
        hyunsik Hyunsik Choi added a comment -

        Committed the patch to master branch. Thanks for the quick review.

        Show
        hyunsik Hyunsik Choi added a comment - Committed the patch to master branch. Thanks for the quick review.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Tajo-master-build #369 (See https://builds.apache.org/job/Tajo-master-build/369/)
        TAJO-1049: Remove the parallel degree limit up to the maximum cluster capacity. (hyunsik: rev e5f7c9591c71c998e702c789d249af20da3eeb6b)

        • CHANGES
        • tajo-core/src/main/java/org/apache/tajo/master/querymaster/SubQuery.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Tajo-master-build #369 (See https://builds.apache.org/job/Tajo-master-build/369/ ) TAJO-1049 : Remove the parallel degree limit up to the maximum cluster capacity. (hyunsik: rev e5f7c9591c71c998e702c789d249af20da3eeb6b) CHANGES tajo-core/src/main/java/org/apache/tajo/master/querymaster/SubQuery.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Tajo-master-CODEGEN-build #11 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/11/)
        TAJO-1049: Remove the parallel degree limit up to the maximum cluster capacity. (hyunsik: rev e5f7c9591c71c998e702c789d249af20da3eeb6b)

        • CHANGES
        • tajo-core/src/main/java/org/apache/tajo/master/querymaster/SubQuery.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Tajo-master-CODEGEN-build #11 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/11/ ) TAJO-1049 : Remove the parallel degree limit up to the maximum cluster capacity. (hyunsik: rev e5f7c9591c71c998e702c789d249af20da3eeb6b) CHANGES tajo-core/src/main/java/org/apache/tajo/master/querymaster/SubQuery.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Tajo-block_iteration-branch-build #4 (See https://builds.apache.org/job/Tajo-block_iteration-branch-build/4/)
        TAJO-1049: Remove the parallel degree limit up to the maximum cluster capacity. (hyunsik: rev e5f7c9591c71c998e702c789d249af20da3eeb6b)

        • CHANGES
        • tajo-core/src/main/java/org/apache/tajo/master/querymaster/SubQuery.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Tajo-block_iteration-branch-build #4 (See https://builds.apache.org/job/Tajo-block_iteration-branch-build/4/ ) TAJO-1049 : Remove the parallel degree limit up to the maximum cluster capacity. (hyunsik: rev e5f7c9591c71c998e702c789d249af20da3eeb6b) CHANGES tajo-core/src/main/java/org/apache/tajo/master/querymaster/SubQuery.java

          People

          • Assignee:
            hyunsik Hyunsik Choi
            Reporter:
            hyunsik Hyunsik Choi
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development