Uploaded image for project: 'Tajo'
  1. Tajo
  2. TAJO-743

Change the default resource allocation policy of leaf tasks

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.0, 0.9.0
    • Fix Version/s: 0.8.0, 0.9.0
    • Component/s: Resource Manager
    • Labels:
      None

      Description

      Currently, resource allocation is calculated by memory base. If a machine have a large memory, in default settings, heavy disk IO per disk is usually caused by high task concurrency. However, it is likely to seem to be problematic.

      When i tested the leaf task scan by 2(concurrency of SATA disk), the performance was better. if you have SAS Storage or SSD, you can increase the disk concurrency. This patch changes the default resource allocation policy to use disk resource.

      The following configs have been available so far:

      • tajo.worker.resource.disks - available disk resource of each worker
      • tajo.task.disk-slot.default - how many disk resource is consumed per task

      Below config is newly introduced in this patch

      • tajo.worker.resource.dfs-dir-aware - it can be true/false. If it is true, each worker uses the number of HDFS datanode's data dirs in the worker as the disk resource. So, tajo.worker.resource.disks is ignored.
      1. TAJO-743.patch
        6 kB
        Jinho Kim
      2. TAJO-743_branch-0.8.0.patch
        6 kB
        Jinho Kim

        Activity

        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Tajo-0.8.0-build #64 (See https://builds.apache.org/job/Tajo-0.8.0-build/64/)
        TAJO-743: Change the default resource allocation policy of leaf tasks. (jinho) (jinossy: rev 6824aab67f1266983810eb3ed4d12102d7aa0b87)

        • tajo-common/src/main/java/org/apache/tajo/conf/TajoConf.java
        • tajo-storage/src/main/java/org/apache/tajo/storage/v2/DiskUtil.java
        • CHANGES.txt
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/TajoResourceAllocator.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/TajoWorker.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Tajo-0.8.0-build #64 (See https://builds.apache.org/job/Tajo-0.8.0-build/64/ ) TAJO-743 : Change the default resource allocation policy of leaf tasks. (jinho) (jinossy: rev 6824aab67f1266983810eb3ed4d12102d7aa0b87) tajo-common/src/main/java/org/apache/tajo/conf/TajoConf.java tajo-storage/src/main/java/org/apache/tajo/storage/v2/DiskUtil.java CHANGES.txt tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/TajoResourceAllocator.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/TajoWorker.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Tajo-master-build #164 (See https://builds.apache.org/job/Tajo-master-build/164/)
        TAJO-743: Change the default resource allocation policy of leaf tasks. (jinho) (jinossy: rev 743c52650d34ed67b777705816a127b7b5f1b0fa)

        • tajo-storage/src/main/java/org/apache/tajo/storage/v2/DiskUtil.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/WorkerHeartbeatService.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/TajoResourceAllocator.java
        • CHANGES.txt
        • tajo-common/src/main/java/org/apache/tajo/conf/TajoConf.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Tajo-master-build #164 (See https://builds.apache.org/job/Tajo-master-build/164/ ) TAJO-743 : Change the default resource allocation policy of leaf tasks. (jinho) (jinossy: rev 743c52650d34ed67b777705816a127b7b5f1b0fa) tajo-storage/src/main/java/org/apache/tajo/storage/v2/DiskUtil.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/WorkerHeartbeatService.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/TajoResourceAllocator.java CHANGES.txt tajo-common/src/main/java/org/apache/tajo/conf/TajoConf.java
        Hide
        jhkim Jinho Kim added a comment -

        This issue got +1 from reviewboard.
        I've just committed it to both master branch and branch-0.8.0.

        Show
        jhkim Jinho Kim added a comment - This issue got +1 from reviewboard. I've just committed it to both master branch and branch-0.8.0.
        Hide
        tajoqa Tajo QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12639360/TAJO-743.patch
        against master revision 631f3d0.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The applied patch does not increase the total number of javadoc warnings.

        +1 checkstyle. The patch generated 0 code style errors.

        -1 findbugs. The patch appears to introduce 204 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in tajo-common tajo-core/tajo-core-backend tajo-storage.

        Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/318//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/318//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core-backend.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/318//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-common.html
        Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/318//console

        This message is automatically generated.

        Show
        tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12639360/TAJO-743.patch against master revision 631f3d0. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The applied patch does not increase the total number of javadoc warnings. +1 checkstyle. The patch generated 0 code style errors. -1 findbugs. The patch appears to introduce 204 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in tajo-common tajo-core/tajo-core-backend tajo-storage. Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/318//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/318//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core-backend.html Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/318//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-common.html Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/318//console This message is automatically generated.
        Hide
        jhkim Jinho Kim added a comment -

        Created a review request against branch master in reviewboard
        https://reviews.apache.org/r/20149/

        Show
        jhkim Jinho Kim added a comment - Created a review request against branch master in reviewboard https://reviews.apache.org/r/20149/

          People

          • Assignee:
            jhkim Jinho Kim
            Reporter:
            jhkim Jinho Kim
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development