Uploaded image for project: 'Tajo'
  1. Tajo
  2. TAJO-588

In some case, leaf task of DefaultTaskScheduler are not distributed execution

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: 0.8.0
    • Component/s: Planner/Optimizer
    • Labels:
      None

      Description

      Leaf task of DefaultTaskScheduler should be fragment size
      If initial request of leaf task is small size, most tasks will be allocate to single node.

      1. TAJO-588_20140206_16:30:08.patch
        7 kB
        Jinho Kim
      2. TAJO-588.patch
        3 kB
        Jinho Kim

        Activity

        Hide
        jhkim Jinho Kim added a comment -

        Created a review request against branch master in reviewboard
        https://reviews.apache.org/r/17743/

        Show
        jhkim Jinho Kim added a comment - Created a review request against branch master in reviewboard https://reviews.apache.org/r/17743/
        Hide
        jhkim Jinho Kim added a comment -

        In addition, this patch fix the invalid task success log

        Show
        jhkim Jinho Kim added a comment - In addition, this patch fix the invalid task success log
        Hide
        tajoqa Tajo QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12627133/TAJO-588.patch
        against master revision 2761436.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The applied patch does not increase the total number of javadoc warnings.

        +1 checkstyle. The patch generated 0 code style errors.

        -1 findbugs. The patch appears to introduce 190 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in tajo-core/tajo-core-backend.

        Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/108//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/108//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core-backend.html
        Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/108//console

        This message is automatically generated.

        Show
        tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12627133/TAJO-588.patch against master revision 2761436. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The applied patch does not increase the total number of javadoc warnings. +1 checkstyle. The patch generated 0 code style errors. -1 findbugs. The patch appears to introduce 190 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in tajo-core/tajo-core-backend. Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/108//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/108//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core-backend.html Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/108//console This message is automatically generated.
        Hide
        jhkim Jinho Kim added a comment -

        Updated the review request against branch master in reviewboard
        https://reviews.apache.org/r/17743/

        Show
        jhkim Jinho Kim added a comment - Updated the review request against branch master in reviewboard https://reviews.apache.org/r/17743/
        Hide
        jhkim Jinho Kim added a comment -

        I've uploaded the second patch that improve initial delay

        Show
        jhkim Jinho Kim added a comment - I've uploaded the second patch that improve initial delay
        Hide
        tajoqa Tajo QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12627308/TAJO-588_20140206_16%3A30%3A08.patch
        against master revision 2761436.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The applied patch does not increase the total number of javadoc warnings.

        +1 checkstyle. The patch generated 0 code style errors.

        -1 findbugs. The patch appears to introduce 190 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in tajo-core/tajo-core-backend.

        Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/109//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/109//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core-backend.html
        Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/109//console

        This message is automatically generated.

        Show
        tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12627308/TAJO-588_20140206_16%3A30%3A08.patch against master revision 2761436. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The applied patch does not increase the total number of javadoc warnings. +1 checkstyle. The patch generated 0 code style errors. -1 findbugs. The patch appears to introduce 190 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in tajo-core/tajo-core-backend. Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/109//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/109//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core-backend.html Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/109//console This message is automatically generated.
        Hide
        jhkim Jinho Kim added a comment -

        This issue got +1 from reviewboard.
        I've just committed it to master branch.

        Show
        jhkim Jinho Kim added a comment - This issue got +1 from reviewboard. I've just committed it to master branch.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Tajo-master-build #58 (See https://builds.apache.org/job/Tajo-master-build/58/)
        TAJO-588: In some case, leaf task of DefaultTaskScheduler are not distributed execution. (jinho) (jinossy: https://git-wip-us.apache.org/repos/asf?p=incubator-tajo.git&a=commit&h=009e0253ca04dd4b353a3f383571123e1e75719c)

        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/DefaultTaskScheduler.java
        • tajo-common/src/main/java/org/apache/tajo/conf/TajoConf.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/SubQuery.java
        • CHANGES.txt
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Tajo-master-build #58 (See https://builds.apache.org/job/Tajo-master-build/58/ ) TAJO-588 : In some case, leaf task of DefaultTaskScheduler are not distributed execution. (jinho) (jinossy: https://git-wip-us.apache.org/repos/asf?p=incubator-tajo.git&a=commit&h=009e0253ca04dd4b353a3f383571123e1e75719c ) tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/DefaultTaskScheduler.java tajo-common/src/main/java/org/apache/tajo/conf/TajoConf.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/SubQuery.java CHANGES.txt

          People

          • Assignee:
            jhkim Jinho Kim
            Reporter:
            jhkim Jinho Kim
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development