Hadoop Common
  1. Hadoop Common
  2. HADOOP-5745

Allow setting the default value of maxRunningJobs for all pools

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      New Fair Scheduler configuration parameter sets a default limit on number of running jobs for all pools.

      Description

      The <pool> element allows setting the maxRunningJobs for that pool. It wold be nice to be able to set a default value for all pools.

      In out configuration, pools are autocreated.. every new uesre gets his own pool. We would like to allow each user to be able to run a max of 5 jobs at a time. For the etl pool, this limit will be set to a greater value,

      1. fairShareMaxJobs.patch
        4 kB
        dhruba borthakur
      2. fairShareMaxJobs.patch
        2 kB
        dhruba borthakur
      3. fairShareMaxJobs.patch
        2 kB
        dhruba borthakur

        Activity

        Hide
        dhruba borthakur added a comment -

        Allow setting the default value for the maxRunningJobs per pool.

        Show
        dhruba borthakur added a comment - Allow setting the default value for the maxRunningJobs per pool.
        Hide
        Matei Zaharia added a comment -

        Dhruba, I think you have a bug in that reloadAllocs doesn't define poolMaxJobsDefault as a local variable. This means that it assigns directly to the class field when it reads that element in the XML file. This is inconsistent with how reloadAllocs handles other settings because it means that if reloadAllocs encounters an error in the file later, it will have set the poolMaxJobsDefault but not the other settings. You just want to define a local variable poolMaxJobsDefault in reloadAllocs the same way it has a local userMaxJobsDefault.

        Show
        Matei Zaharia added a comment - Dhruba, I think you have a bug in that reloadAllocs doesn't define poolMaxJobsDefault as a local variable. This means that it assigns directly to the class field when it reads that element in the XML file. This is inconsistent with how reloadAllocs handles other settings because it means that if reloadAllocs encounters an error in the file later, it will have set the poolMaxJobsDefault but not the other settings. You just want to define a local variable poolMaxJobsDefault in reloadAllocs the same way it has a local userMaxJobsDefault.
        Hide
        dhruba borthakur added a comment -

        Thanks for the review Matei. Please let me know if this patch looks good.

        Show
        dhruba borthakur added a comment - Thanks for the review Matei. Please let me know if this patch looks good.
        Hide
        Matei Zaharia added a comment -

        +1 Looks good to me. I'll wait for Hudson to run and commit it.

        Show
        Matei Zaharia added a comment - +1 Looks good to me. I'll wait for Hudson to run and commit it.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12406574/fairShareMaxJobs.patch
        against trunk revision 770321.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no tests are needed for this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/267/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/267/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/267/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/267/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12406574/fairShareMaxJobs.patch against trunk revision 770321. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/267/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/267/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/267/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/267/console This message is automatically generated.
        Hide
        dhruba borthakur added a comment -

        have to write unit test.

        Show
        dhruba borthakur added a comment - have to write unit test.
        Hide
        dhruba borthakur added a comment -

        Added unit test.

        Show
        dhruba borthakur added a comment - Added unit test.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12407129/fairShareMaxJobs.patch
        against trunk revision 771179.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/285/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/285/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/285/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/285/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12407129/fairShareMaxJobs.patch against trunk revision 771179. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/285/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/285/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/285/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/285/console This message is automatically generated.
        Hide
        dhruba borthakur added a comment -

        It is imposible that the unit test that failed (testDFSRestart) is caused by this fix in the fair-share scheduler.

        Show
        dhruba borthakur added a comment - It is imposible that the unit test that failed (testDFSRestart) is caused by this fix in the fair-share scheduler.
        Hide
        Matei Zaharia added a comment -

        I committed this. Thanks Dhruba.

        Show
        Matei Zaharia added a comment - I committed this. Thanks Dhruba.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-trunk #827 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/827/)
        . Allow setting the default value of maxRunningJobs for all
        pools. Contributed by Dhruba Borthakur.

        Show
        Hudson added a comment - Integrated in Hadoop-trunk #827 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/827/ ) . Allow setting the default value of maxRunningJobs for all pools. Contributed by Dhruba Borthakur.
        Hide
        Robert Chansler added a comment -

        Editorial pass over all release notes prior to publication of 0.21.

        Show
        Robert Chansler added a comment - Editorial pass over all release notes prior to publication of 0.21.

          People

          • Assignee:
            dhruba borthakur
            Reporter:
            dhruba borthakur
          • Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development