Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-944

Extend FairShare scheduler to fair-share memory usage in the cluster

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: contrib/fair-share
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The FairShare Scheduler has an extensible LoadManager API to regulate allocating new tasks on a particular TaskTracker. In similar lines, it would be nice if the FairShare Scheduler can have a pluggable policy to regulate new tasks from a particular job. This will allow one to skip scheduling tasks of a job that is eating a large percentage of memory in the cluster, i.e. fair-share of memory resources among jobs.

      1. LoadManager2.txt
        5 kB
        dhruba borthakur
      2. LoadManager.txt
        4 kB
        dhruba borthakur

        Issue Links

          Activity

          Hide
          dhruba borthakur added a comment -

          One simple way to do this would be to pass in the LoadManager object into the Schedulable.assignTask().

          Show
          dhruba borthakur added a comment - One simple way to do this would be to pass in the LoadManager object into the Schedulable.assignTask().
          Hide
          Matei Zaharia added a comment -

          I think that rather than passing a LoadManager to the Schedulables and having them check whether they can run, it's better design to put all the logic in the LoadManager and ask it whether it's okay to run a particular job. I'd do this the following way:

          • LoadManager contains a method called canLaunchTask(JobInProgress job, TaskType type, TaskTracker Status tt)
          • In JobSchedulable.assignTasks, before looking for a task, the JobSchedulable first checks whether scheduler.getLoadManager().canLaunchTask(job, type, tt) is true. If it isn't,
          • The default implementation of canLaunchTask always returns true, but in the memory-aware LoadManager, it can be made to return false.
          Show
          Matei Zaharia added a comment - I think that rather than passing a LoadManager to the Schedulables and having them check whether they can run, it's better design to put all the logic in the LoadManager and ask it whether it's okay to run a particular job. I'd do this the following way: LoadManager contains a method called canLaunchTask(JobInProgress job, TaskType type, TaskTracker Status tt) In JobSchedulable.assignTasks, before looking for a task, the JobSchedulable first checks whether scheduler.getLoadManager().canLaunchTask(job, type, tt) is true. If it isn't, The default implementation of canLaunchTask always returns true, but in the memory-aware LoadManager, it can be made to return false.
          Hide
          dhruba borthakur added a comment -

          +1 to Matie's proposal.

          Show
          dhruba borthakur added a comment - +1 to Matie's proposal.
          Hide
          dhruba borthakur added a comment -

          Implemented Matei's proposal. I think the current unit tests will test that this code refactoring does not break existing functionality.

          Show
          dhruba borthakur added a comment - Implemented Matei's proposal. I think the current unit tests will test that this code refactoring does not break existing functionality.
          Hide
          Matei Zaharia added a comment -

          +1 looks good to me, except for one minor detail: I'd change the JavaDoc on canLaunchTask from "Can a given

          {@link TaskTracker} run another new task?" to "Can a given {@link TaskTracker}

          run another new task from a given job? This method is provided for use by LoadManagers that take into account jobs' individual resource needs when placing tasks." This way the purpose of the method, and its difference from canAssignMap/canAssignReduce, is better explained. It might also be good to look at the comments for canAssignMap/Reduce and make them say that these methods are used to check whether any attempt to find a job with maps/reduces should be made.

          Show
          Matei Zaharia added a comment - +1 looks good to me, except for one minor detail: I'd change the JavaDoc on canLaunchTask from "Can a given {@link TaskTracker} run another new task?" to "Can a given {@link TaskTracker} run another new task from a given job? This method is provided for use by LoadManagers that take into account jobs' individual resource needs when placing tasks." This way the purpose of the method, and its difference from canAssignMap/canAssignReduce, is better explained. It might also be good to look at the comments for canAssignMap/Reduce and make them say that these methods are used to check whether any attempt to find a job with maps/reduces should be made.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          I see in the patch attached that only one concrete implementation CapBasedLoadManager is done for the LoadManager which in turn doesn't take into account any resource usage. I guess you are planning a proper implementation for this feature regarding fair-share of memory usage in another JIRA.

          Some points still not dealt with in this JIRA. I bring about these points so as to know if you are thinking or have already thought anything about this.

          • Job configuration about how users specify the resource usage. Some memory related configuration properties are added to the framework while working for memory monitoring on TTs as well as memory usage based scheduling in CapacityTaskScheduler. You may want to reuse some/all of it.
          • Capturing the scheduling decisions involved when we are not able to find a task from a Schedulable because of lack of resources on a given TaskTasker.

          Regarding the latter, the current patch just returns null, which is similar to the decision CapacityTaskScheduler used to take in previous versions - i.e. block the TT till it can be given a task from the job at the head of the queue/pool. Sometime back, we investigated how this approach works with FairScheduler and realized some important implications. For e.g, because the order of jobs might change significantly in consecutive iterations of FairScheduler, just returning null may not work at all. Eventually we may end up waiting for a long time if significant number of jobs ask for high amount of resources.

          Thoughts?

          Show
          Vinod Kumar Vavilapalli added a comment - I see in the patch attached that only one concrete implementation CapBasedLoadManager is done for the LoadManager which in turn doesn't take into account any resource usage. I guess you are planning a proper implementation for this feature regarding fair-share of memory usage in another JIRA. Some points still not dealt with in this JIRA. I bring about these points so as to know if you are thinking or have already thought anything about this. Job configuration about how users specify the resource usage. Some memory related configuration properties are added to the framework while working for memory monitoring on TTs as well as memory usage based scheduling in CapacityTaskScheduler. You may want to reuse some/all of it. Capturing the scheduling decisions involved when we are not able to find a task from a Schedulable because of lack of resources on a given TaskTasker. Regarding the latter, the current patch just returns null, which is similar to the decision CapacityTaskScheduler used to take in previous versions - i.e. block the TT till it can be given a task from the job at the head of the queue/pool. Sometime back, we investigated how this approach works with FairScheduler and realized some important implications. For e.g, because the order of jobs might change significantly in consecutive iterations of FairScheduler, just returning null may not work at all. Eventually we may end up waiting for a long time if significant number of jobs ask for high amount of resources. Thoughts?
          Hide
          dhruba borthakur added a comment -

          Incorporated Matie's review comments.

          Vinod: The goal that we have in mind is slightly different from what the capacity scheduler has done (pl correct me if I am wrong). Unlike the capacity scheduler, there is no assumption that the user knows (upfront, before submitting job) how much memory/CPU/network that job will need. A realtime stream of resource usage will be fed into the new LoadManager that can then dynamically decide whether to run another task on that machine or not. Your feedback and experience vis-a-vis that Capacity scheduler is very valuable here... let's continue the conversation via MAPREDUCE-961

          Show
          dhruba borthakur added a comment - Incorporated Matie's review comments. Vinod: The goal that we have in mind is slightly different from what the capacity scheduler has done (pl correct me if I am wrong). Unlike the capacity scheduler, there is no assumption that the user knows (upfront, before submitting job) how much memory/CPU/network that job will need. A realtime stream of resource usage will be fed into the new LoadManager that can then dynamically decide whether to run another task on that machine or not. Your feedback and experience vis-a-vis that Capacity scheduler is very valuable here... let's continue the conversation via MAPREDUCE-961
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12418880/LoadManager2.txt
          against trunk revision 812287.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/44/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/44/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/44/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/44/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12418880/LoadManager2.txt against trunk revision 812287. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/44/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/44/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/44/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/44/console This message is automatically generated.
          Hide
          Scott Chen added a comment -

          I wonder where why there are test errors. canLaunchTask only returns true so it shouldn't change anything. I have applied this patch and run ant clean test. It shows no test error.

          Show
          Scott Chen added a comment - I wonder where why there are test errors. canLaunchTask only returns true so it shouldn't change anything. I have applied this patch and run ant clean test. It shows no test error.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          The test failures were due to HADOOP-6243, which is fixed now. Resubmitting to Hudson should do.

          Show
          Vinod Kumar Vavilapalli added a comment - The test failures were due to HADOOP-6243 , which is fixed now. Resubmitting to Hudson should do.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12418880/LoadManager2.txt
          against trunk revision 812546.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/52/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/52/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/52/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/52/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12418880/LoadManager2.txt against trunk revision 812546. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/52/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/52/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/52/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/52/console This message is automatically generated.
          Hide
          Konstantin Boudnik added a comment -

          It'd be good to have a note on why there's no new tests added within that fix.

          Show
          Konstantin Boudnik added a comment - It'd be good to have a note on why there's no new tests added within that fix.
          Hide
          dhruba borthakur added a comment -

          This is not really a bug fix. It is mostly a refactoring of the code so that a certain functionality is separated out more cleanly. HADOOP-961 will benefit from this JIRA and it will have quite a few unit tests associated with it.

          Show
          dhruba borthakur added a comment - This is not really a bug fix. It is mostly a refactoring of the code so that a certain functionality is separated out more cleanly. HADOOP-961 will benefit from this JIRA and it will have quite a few unit tests associated with it.
          Hide
          dhruba borthakur added a comment -

          I just committed this.

          Show
          dhruba borthakur added a comment - I just committed this.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #28 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/28/)
          . Extend the LoadManager API of the fair-share scheduler
          to support regulating tasks for a job based on resources currently in use
          by that job. (dhruba)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #28 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/28/ ) . Extend the LoadManager API of the fair-share scheduler to support regulating tasks for a job based on resources currently in use by that job. (dhruba)

            People

            • Assignee:
              dhruba borthakur
              Reporter:
              dhruba borthakur
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development