Uploaded image for project: 'Tajo'
  1. Tajo
  2. TAJO-647

Work unbalance on disk scheduling of DefaultScheduler

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: 0.8.0, 0.9.0
    • Component/s: Resource Manager
    • Labels:
      None

      Description

      The main problem is that localAllocation does not find next lowest volume. and it will assign to remote in rack.
      We should control the remote tasks.

      • Leaf scheduling priorities
        1. tasks in a volume of host
        2. unknown disk, non-splitable in local host
        3. remote tasks in rack. (consider the remaining tasks)
        4. random remote tasks. (tail concurrency control of scheduled tasks)
      1. TAJO-647_140312_145207.patch
        28 kB
        Hyunsik Choi
      2. TAJO-647_20140307_14:36:35.patch
        25 kB
        Jinho Kim
      3. TAJO-647_20140313_17:53:23.patch
        29 kB
        Jinho Kim
      4. TAJO-647.patch
        21 kB
        Jinho Kim

        Issue Links

          Activity

          Hide
          jhkim Jinho Kim added a comment - - edited

          Created a review request against branch master in reviewboard
          https://reviews.apache.org/r/18728

          Show
          jhkim Jinho Kim added a comment - - edited Created a review request against branch master in reviewboard https://reviews.apache.org/r/18728
          Hide
          tajoqa Tajo QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12632442/TAJO-647.patch
          against master revision f4a9fd1.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 javadoc. The applied patch generated 49 javadoc warnings (more than the trunk's current 48 warnings).

          +1 checkstyle. The patch generated 0 code style errors.

          -1 findbugs. The patch appears to introduce 183 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in tajo-core/tajo-core-backend.

          Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/177//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/177//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core-backend.html
          Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/177//console

          This message is automatically generated.

          Show
          tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12632442/TAJO-647.patch against master revision f4a9fd1. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 javadoc. The applied patch generated 49 javadoc warnings (more than the trunk's current 48 warnings). +1 checkstyle. The patch generated 0 code style errors. -1 findbugs. The patch appears to introduce 183 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in tajo-core/tajo-core-backend. Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/177//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/177//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core-backend.html Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/177//console This message is automatically generated.
          Hide
          jhkim Jinho Kim added a comment -

          Updated the review request against branch master in reviewboard
          https://reviews.apache.org/r/18728/

          Show
          jhkim Jinho Kim added a comment - Updated the review request against branch master in reviewboard https://reviews.apache.org/r/18728/
          Hide
          tajoqa Tajo QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12633316/TAJO-647_20140307_14%3A36%3A35.patch
          against master revision e3da0ca.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 javadoc. The applied patch generated 49 javadoc warnings (more than the trunk's current 48 warnings).

          +1 checkstyle. The patch generated 0 code style errors.

          -1 findbugs. The patch appears to introduce 183 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in tajo-core/tajo-core-backend.

          Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/203//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/203//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core-backend.html
          Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/203//console

          This message is automatically generated.

          Show
          tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12633316/TAJO-647_20140307_14%3A36%3A35.patch against master revision e3da0ca. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 javadoc. The applied patch generated 49 javadoc warnings (more than the trunk's current 48 warnings). +1 checkstyle. The patch generated 0 code style errors. -1 findbugs. The patch appears to introduce 183 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in tajo-core/tajo-core-backend. Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/203//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/203//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core-backend.html Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/203//console This message is automatically generated.
          Hide
          hyunsik Hyunsik Choi added a comment -

          Updated the review request against branch master in reviewboard
          https://reviews.apache.org/r/18728/

          Show
          hyunsik Hyunsik Choi added a comment - Updated the review request against branch master in reviewboard https://reviews.apache.org/r/18728/
          Hide
          jhkim Jinho Kim added a comment -

          Updated the review request against branch master in reviewboard
          https://reviews.apache.org/r/18728/

          Show
          jhkim Jinho Kim added a comment - Updated the review request against branch master in reviewboard https://reviews.apache.org/r/18728/
          Hide
          tajoqa Tajo QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12634402/TAJO-647_20140313_17%3A53%3A23.patch
          against master revision 44b28e6.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 javadoc. The applied patch generated 49 javadoc warnings (more than the trunk's current 48 warnings).

          +1 checkstyle. The patch generated 0 code style errors.

          -1 findbugs. The patch appears to introduce 183 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in tajo-core/tajo-core-backend.

          Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/212//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/212//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core-backend.html
          Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/212//console

          This message is automatically generated.

          Show
          tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12634402/TAJO-647_20140313_17%3A53%3A23.patch against master revision 44b28e6. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 javadoc. The applied patch generated 49 javadoc warnings (more than the trunk's current 48 warnings). +1 checkstyle. The patch generated 0 code style errors. -1 findbugs. The patch appears to introduce 183 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in tajo-core/tajo-core-backend. Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/212//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/212//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core-backend.html Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/212//console This message is automatically generated.
          Hide
          jhkim Jinho Kim added a comment -

          This issue got +1 from reviewboard.
          I've just committed it to both master branch and branch-0.8.0.

          Show
          jhkim Jinho Kim added a comment - This issue got +1 from reviewboard. I've just committed it to both master branch and branch-0.8.0.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Tajo-0.8.0-build #22 (See https://builds.apache.org/job/Tajo-0.8.0-build/22/)
          TAJO-647: Work unbalance on disk scheduling of DefaultScheduler. (jinho) (jinossy: https://git-wip-us.apache.org/repos/asf/?p=incubator-tajo.git&a=commit&h=9d4965a34a8a620ae2cf6d6a1dd2ff69c0418269)

          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/SubQuery.java
          • CHANGES.txt
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/DefaultTaskScheduler.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryUnit.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Tajo-0.8.0-build #22 (See https://builds.apache.org/job/Tajo-0.8.0-build/22/ ) TAJO-647 : Work unbalance on disk scheduling of DefaultScheduler. (jinho) (jinossy: https://git-wip-us.apache.org/repos/asf/?p=incubator-tajo.git&a=commit&h=9d4965a34a8a620ae2cf6d6a1dd2ff69c0418269 ) tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/SubQuery.java CHANGES.txt tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/DefaultTaskScheduler.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryUnit.java
          Hide
          hudson Hudson added a comment -

          ABORTED: Integrated in Tajo-master-build #118 (See https://builds.apache.org/job/Tajo-master-build/118/)
          TAJO-647: Work unbalance on disk scheduling of DefaultScheduler. (jinho) (jinossy: https://git-wip-us.apache.org/repos/asf/?p=incubator-tajo.git&a=commit&h=54bf9a7c8b7851381c94661500ce329f2c12e9ba)

          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/SubQuery.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/DefaultTaskScheduler.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryUnit.java
          • CHANGES.txt
          Show
          hudson Hudson added a comment - ABORTED: Integrated in Tajo-master-build #118 (See https://builds.apache.org/job/Tajo-master-build/118/ ) TAJO-647 : Work unbalance on disk scheduling of DefaultScheduler. (jinho) (jinossy: https://git-wip-us.apache.org/repos/asf/?p=incubator-tajo.git&a=commit&h=54bf9a7c8b7851381c94661500ce329f2c12e9ba ) tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/SubQuery.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/DefaultTaskScheduler.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryUnit.java CHANGES.txt

            People

            • Assignee:
              jhkim Jinho Kim
              Reporter:
              jhkim Jinho Kim
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development