Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-698

Per-pool task limits for the fair scheduler

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: contrib/fair-share
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Per-pool map and reduce caps for Fair Scheduler.

      Description

      The fair scheduler could use a way to cap the share of a given pool similar to MAPREDUCE-532.

      1. mapreduce-698-trunk-5.patch
        15 kB
        Matei Zaharia
      2. mapreduce-698-trunk-4.patch
        15 kB
        Kevin Peterson
      3. mapreduce-698-trunk-3.patch
        12 kB
        Kevin Peterson
      4. mapreduce-698-trunk.patch
        14 kB
        Kevin Peterson
      5. mapreduce-698-trunk.patch
        11 kB
        Kevin Peterson
      6. MAPREDUCE-698-prelim.patch
        8 kB
        Kevin Peterson

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #196 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/196/)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #196 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/196/ )
          Hide
          Matei Zaharia added a comment -

          The test failures are not caused by this patch, so I've committed it. Thanks, Kevin!

          Show
          Matei Zaharia added a comment - The test failures are not caused by this patch, so I've committed it. Thanks, Kevin!
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12428317/mapreduce-698-trunk-5.patch
          against trunk revision 891920.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/214/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/214/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/214/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/214/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12428317/mapreduce-698-trunk-5.patch against trunk revision 891920. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/214/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/214/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/214/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/214/console This message is automatically generated.
          Hide
          Matei Zaharia added a comment -

          Here's the patch with the for loop removed. I'm going to run it through Hudson for good measure, but it seems to be working fine from my point of view, and the test failures in the previous run were unrelated. I'll commit it unless Hudson complains.

          Show
          Matei Zaharia added a comment - Here's the patch with the for loop removed. I'm going to run it through Hudson for good measure, but it seems to be working fine from my point of view, and the test failures in the previous run were unrelated. I'll commit it unless Hudson complains.
          Hide
          Kevin Peterson added a comment -

          That was for checking if the allocations were consistent (min < max), I moved this into the loop where they read but missed this bit it looks like.

          Show
          Kevin Peterson added a comment - That was for checking if the allocations were consistent (min < max), I moved this into the loop where they read but missed this bit it looks like.
          Hide
          Matei Zaharia added a comment -

          I've looked at the patch more carefully, and it all looks good, except there seems to be a loop doing nothing in PoolManager:

          +    for(String pool : poolNamesInAllocFile) {
          +    }
          

          I can remove this myself and commit the patch, unless there was a reason you had it there (and forgot to put in some code).

          Show
          Matei Zaharia added a comment - I've looked at the patch more carefully, and it all looks good, except there seems to be a loop doing nothing in PoolManager: + for(String pool : poolNamesInAllocFile) { + } I can remove this myself and commit the patch, unless there was a reason you had it there (and forgot to put in some code).
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12427904/mapreduce-698-trunk-4.patch
          against trunk revision 890983.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/196/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/196/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/196/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/196/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12427904/mapreduce-698-trunk-4.patch against trunk revision 890983. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/196/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/196/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/196/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/196/console This message is automatically generated.
          Hide
          Matei Zaharia added a comment -

          Resubmitting patch to Hudson, since the last test run seemed to be very broken.

          The changes look good as far as I can tell!

          Show
          Matei Zaharia added a comment - Resubmitting patch to Hudson, since the last test run seemed to be very broken. The changes look good as far as I can tell!
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12427904/mapreduce-698-trunk-4.patch
          against trunk revision 889786.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/191/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/191/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/191/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/191/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12427904/mapreduce-698-trunk-4.patch against trunk revision 889786. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/191/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/191/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/191/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/191/console This message is automatically generated.
          Hide
          Kevin Peterson added a comment -

          Should be ready to go.

          Show
          Kevin Peterson added a comment - Should be ready to go.
          Hide
          Kevin Peterson added a comment -

          cancel and resubmit to pick up newer patch.

          Show
          Kevin Peterson added a comment - cancel and resubmit to pick up newer patch.
          Hide
          Kevin Peterson added a comment -

          Added some warnings if it's configured with max < min, fixed layout in servlet to match current version.

          Also, I believe the tests have been fixed in trunk, so this should pass release audit now.

          Show
          Kevin Peterson added a comment - Added some warnings if it's configured with max < min, fixed layout in servlet to match current version. Also, I believe the tests have been fixed in trunk, so this should pass release audit now.
          Hide
          Matei Zaharia added a comment -

          The patch looks good to me. The only other thing I'd ask for is some documentation of what happens if a pool's max share is set to lower than its min share. The best way to do this might be to log a warning using LOG.warn before the "commit" part of PoolManager.reloadAllocs and say that when a pool's max share is lower than its min share, the max share takes precedence. You could also display this warning on the web UI when pools with max share < min share exist, because admins are likely to be looking at the UI after they modify their config file.

          The unit test failures seem to be unrelated to the patch - in particular, the one in the fair scheduler is due to MAPREDUCE-1245.

          Show
          Matei Zaharia added a comment - The patch looks good to me. The only other thing I'd ask for is some documentation of what happens if a pool's max share is set to lower than its min share. The best way to do this might be to log a warning using LOG.warn before the "commit" part of PoolManager.reloadAllocs and say that when a pool's max share is lower than its min share, the max share takes precedence. You could also display this warning on the web UI when pools with max share < min share exist, because admins are likely to be looking at the UI after they modify their config file. The unit test failures seem to be unrelated to the patch - in particular, the one in the fair scheduler is due to MAPREDUCE-1245 .
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12426456/mapreduce-698-trunk-3.patch
          against trunk revision 885530.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/156/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/156/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/156/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/156/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12426456/mapreduce-698-trunk-3.patch against trunk revision 885530. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/156/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/156/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/156/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/156/console This message is automatically generated.
          Hide
          Kevin Peterson added a comment -

          changes from previous patch:

          • extra newline in fairscheduler.java removed
          • removed "single test" changes from build-contrib.xml (they didn't accomplish what I wanted – to run just a single test method)
          • Regarding checkAssignment, I made the change you suggested, but I'm not sure I'm testing things in the best way. The only thing I'm concerned with is that it ends up scheduling the right number from each pool, the only way I was able to get it to actually assign the jobs was to use checkAssignment.
          • in the UI, labels are "Max Share"
          • Removed Pool.numRunningTasks since it was only used from within PoolSchedulable, where this data is already available.
          • Moved cap from getDemand() to updateDemand().
          • Documentation updated
          • Removed tabs.
          Show
          Kevin Peterson added a comment - changes from previous patch: extra newline in fairscheduler.java removed removed "single test" changes from build-contrib.xml (they didn't accomplish what I wanted – to run just a single test method) Regarding checkAssignment, I made the change you suggested, but I'm not sure I'm testing things in the best way. The only thing I'm concerned with is that it ends up scheduling the right number from each pool, the only way I was able to get it to actually assign the jobs was to use checkAssignment. in the UI, labels are "Max Share" Removed Pool.numRunningTasks since it was only used from within PoolSchedulable, where this data is already available. Moved cap from getDemand() to updateDemand(). Documentation updated Removed tabs.
          Hide
          Matei Zaharia added a comment -

          Oops, I forgot to add one other important comment: The new config parameters should be documented in the fair scheduler's Forrest documentation in src/docs/src/documentation/content/xdocs/fair_scheduler.xml (in the Allocation File section). Also, the documentation should say what happens if the maxTasks of a pool is set lower than its minTasks: in this case, the maxTasks takes precedence. It might also be good to print a warning in PoolManager when loading a config file if we see a pool with maxTasks < minTasks. You should do this after you finish reading the a <pool> element.

          One other very minor thing - there seem to be some tabs in the patch, replace them with spaces.

          Thanks for taking the time to port this to trunk!

          Show
          Matei Zaharia added a comment - Oops, I forgot to add one other important comment: The new config parameters should be documented in the fair scheduler's Forrest documentation in src/docs/src/documentation/content/xdocs/fair_scheduler.xml (in the Allocation File section). Also, the documentation should say what happens if the maxTasks of a pool is set lower than its minTasks: in this case, the maxTasks takes precedence. It might also be good to print a warning in PoolManager when loading a config file if we see a pool with maxTasks < minTasks. You should do this after you finish reading the a <pool> element. One other very minor thing - there seem to be some tabs in the patch, replace them with spaces. Thanks for taking the time to port this to trunk!
          Hide
          Matei Zaharia added a comment -

          The patch mostly looks good to me, but I have a few comments:

          • There seem to be some extra newlines added in FairScheduler.java. You might want to just svn revert it.
          • What is the purpose of the changes to build-contrib.xml? Are they just something you copied in from another build.xml file?
          • In the unit test, it would be good to submit the jobs first (with an advanceTime between them) before doing any checkAssignment's, so that both jobs are initially available. Note that the scheduler will probably alternate between assigning tasks from the two jobs in this case, but it should still not let job 1 go over its max share.
          • In the web UI, can you call the new "Max" column "Max Share" instead, so it is more consistent with the other column names?
          • You can simplify Pool.numRunningTasks(TaskType type) to just return getSchedulable(type).getRunningTasks(). It might also be good to rename the method to getRunningTasks instead of nunRunningTasks so that it's more consistent with the rest of the code.
          • Instead of making PoolSchedulable.getDemand() return maxTasks if demand > maxTasks, it would be better to change PoolSchedulable.updateDemand() to cap the demand at the end of the method, so that getDemand() just returns demand. Otherwise, it will be a little confusing to have a variable called demand in PoolSchedulable whose value is not the same as that returned by getDemand().
          Show
          Matei Zaharia added a comment - The patch mostly looks good to me, but I have a few comments: There seem to be some extra newlines added in FairScheduler.java. You might want to just svn revert it. What is the purpose of the changes to build-contrib.xml? Are they just something you copied in from another build.xml file? In the unit test, it would be good to submit the jobs first (with an advanceTime between them) before doing any checkAssignment's, so that both jobs are initially available. Note that the scheduler will probably alternate between assigning tasks from the two jobs in this case, but it should still not let job 1 go over its max share. In the web UI, can you call the new "Max" column "Max Share" instead, so it is more consistent with the other column names? You can simplify Pool.numRunningTasks(TaskType type) to just return getSchedulable(type).getRunningTasks(). It might also be good to rename the method to getRunningTasks instead of nunRunningTasks so that it's more consistent with the rest of the code. Instead of making PoolSchedulable.getDemand() return maxTasks if demand > maxTasks, it would be better to change PoolSchedulable.updateDemand() to cap the demand at the end of the method, so that getDemand() just returns demand. Otherwise, it will be a little confusing to have a variable called demand in PoolSchedulable whose value is not the same as that returned by getDemand().
          Hide
          dhruba borthakur added a comment -

          it would be nice if somebody can review this patch.

          Show
          dhruba borthakur added a comment - it would be nice if somebody can review this patch.
          Hide
          Kevin Peterson added a comment -

          Update patch to display caps in the scheduler UI.

          Show
          Kevin Peterson added a comment - Update patch to display caps in the scheduler UI.
          Hide
          dhruba borthakur added a comment -

          Amazing Kevin, thanks a lot. I will wait for your next version of the patch that includes the display in the UI.

          Show
          dhruba borthakur added a comment - Amazing Kevin, thanks a lot. I will wait for your next version of the patch that includes the display in the UI.
          Hide
          Kevin Peterson added a comment -

          Patch relative to head. Overrides demand() and assign(), unit test included.

          I'm able to run the test I added locally, but I'm having trouble with some other test when I run the suite. I haven't tested this on a real cluster, just in unit tests.

          Still need to add the display in scheduler servlet. It's sitting on my machine at work.

          Show
          Kevin Peterson added a comment - Patch relative to head. Overrides demand() and assign(), unit test included. I'm able to run the test I added locally, but I'm having trouble with some other test when I run the suite. I haven't tested this on a real cluster, just in unit tests. Still need to add the display in scheduler servlet. It's sitting on my machine at work.
          Hide
          dhruba borthakur added a comment -

          Hi Matie, if you could review the patch for 0.20, that will be great too.

          Show
          dhruba borthakur added a comment - Hi Matie, if you could review the patch for 0.20, that will be great too.
          Hide
          dhruba borthakur added a comment -

          Hi Kevin, will it be possible for you to upload a patch for trunk? We are definitely interested in trying it out. Thanks.

          Show
          dhruba borthakur added a comment - Hi Kevin, will it be possible for you to upload a patch for trunk? We are definitely interested in trying it out. Thanks.
          Hide
          Matei Zaharia added a comment -

          Hi Kevin,

          The patch looks reasonable for 0.20, though I haven't had a chance to look at it in detail yet. For trunk, MAPREDUCE-706 should have made your life easier; you should be able to just update the demand() method in PoolSchedulable to return the min of the pool's maxMaps (or maxReduces) and the sum of its jobs' demands. Let me know if you have trouble for that. In trunk there's also a fairly detailed design doc for the fair scheduler that should be helpful.

          Matei

          Show
          Matei Zaharia added a comment - Hi Kevin, The patch looks reasonable for 0.20, though I haven't had a chance to look at it in detail yet. For trunk, MAPREDUCE-706 should have made your life easier; you should be able to just update the demand() method in PoolSchedulable to return the min of the pool's maxMaps (or maxReduces) and the sum of its jobs' demands. Let me know if you have trouble for that. In trunk there's also a fairly detailed design doc for the fair scheduler that should be helpful. Matei
          Hide
          Kevin Peterson added a comment -

          Attached patch works for me, but I think I need to make some changes for interaction with the multiple-assign before it's ready to commit. Patch is relative to 0.20 branch.

          It allows two new fields in the pools definition, "maxMaps" and "maxReduces".

          My need for this is interacting with outside services (in this case, an HBase cluster that's significantly smaller than my Hadoop cluster).

          Show
          Kevin Peterson added a comment - Attached patch works for me, but I think I need to make some changes for interaction with the multiple-assign before it's ready to commit. Patch is relative to 0.20 branch. It allows two new fields in the pools definition, "maxMaps" and "maxReduces". My need for this is interacting with outside services (in this case, an HBase cluster that's significantly smaller than my Hadoop cluster).

            People

            • Assignee:
              Kevin Peterson
              Reporter:
              Matei Zaharia
            • Votes:
              1 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development