Hadoop Common
  1. Hadoop Common
  2. HADOOP-5745

Allow setting the default value of maxRunningJobs for all pools

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      New Fair Scheduler configuration parameter sets a default limit on number of running jobs for all pools.

      Description

      The <pool> element allows setting the maxRunningJobs for that pool. It wold be nice to be able to set a default value for all pools.

      In out configuration, pools are autocreated.. every new uesre gets his own pool. We would like to allow each user to be able to run a max of 5 jobs at a time. For the etl pool, this limit will be set to a greater value,

      1. fairShareMaxJobs.patch
        2 kB
        dhruba borthakur
      2. fairShareMaxJobs.patch
        2 kB
        dhruba borthakur
      3. fairShareMaxJobs.patch
        4 kB
        dhruba borthakur

        Activity

        dhruba borthakur created issue -
        Hide
        dhruba borthakur added a comment -

        Allow setting the default value for the maxRunningJobs per pool.

        Show
        dhruba borthakur added a comment - Allow setting the default value for the maxRunningJobs per pool.
        dhruba borthakur made changes -
        Field Original Value New Value
        Attachment fairShareMaxJobs.patch [ 12406460 ]
        Hide
        Matei Zaharia added a comment -

        Dhruba, I think you have a bug in that reloadAllocs doesn't define poolMaxJobsDefault as a local variable. This means that it assigns directly to the class field when it reads that element in the XML file. This is inconsistent with how reloadAllocs handles other settings because it means that if reloadAllocs encounters an error in the file later, it will have set the poolMaxJobsDefault but not the other settings. You just want to define a local variable poolMaxJobsDefault in reloadAllocs the same way it has a local userMaxJobsDefault.

        Show
        Matei Zaharia added a comment - Dhruba, I think you have a bug in that reloadAllocs doesn't define poolMaxJobsDefault as a local variable. This means that it assigns directly to the class field when it reads that element in the XML file. This is inconsistent with how reloadAllocs handles other settings because it means that if reloadAllocs encounters an error in the file later, it will have set the poolMaxJobsDefault but not the other settings. You just want to define a local variable poolMaxJobsDefault in reloadAllocs the same way it has a local userMaxJobsDefault.
        Hide
        dhruba borthakur added a comment -

        Thanks for the review Matei. Please let me know if this patch looks good.

        Show
        dhruba borthakur added a comment - Thanks for the review Matei. Please let me know if this patch looks good.
        dhruba borthakur made changes -
        Attachment fairShareMaxJobs.patch [ 12406574 ]
        dhruba borthakur made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Matei Zaharia added a comment -

        +1 Looks good to me. I'll wait for Hudson to run and commit it.

        Show
        Matei Zaharia added a comment - +1 Looks good to me. I'll wait for Hudson to run and commit it.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12406574/fairShareMaxJobs.patch
        against trunk revision 770321.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no tests are needed for this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/267/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/267/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/267/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/267/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12406574/fairShareMaxJobs.patch against trunk revision 770321. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/267/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/267/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/267/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/267/console This message is automatically generated.
        Hide
        dhruba borthakur added a comment -

        have to write unit test.

        Show
        dhruba borthakur added a comment - have to write unit test.
        dhruba borthakur made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Hide
        dhruba borthakur added a comment -

        Added unit test.

        Show
        dhruba borthakur added a comment - Added unit test.
        dhruba borthakur made changes -
        Attachment fairShareMaxJobs.patch [ 12407129 ]
        dhruba borthakur made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12407129/fairShareMaxJobs.patch
        against trunk revision 771179.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/285/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/285/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/285/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/285/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12407129/fairShareMaxJobs.patch against trunk revision 771179. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/285/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/285/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/285/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/285/console This message is automatically generated.
        Hide
        dhruba borthakur added a comment -

        It is imposible that the unit test that failed (testDFSRestart) is caused by this fix in the fair-share scheduler.

        Show
        dhruba borthakur added a comment - It is imposible that the unit test that failed (testDFSRestart) is caused by this fix in the fair-share scheduler.
        Matei Zaharia made changes -
        Hadoop Flags [Reviewed]
        Release Note Added a configuration parameter for setting a default limit on number of running jobs for all pools in the fair scheduler.
        Issue Type Improvement [ 4 ] New Feature [ 2 ]
        Hide
        Matei Zaharia added a comment -

        I committed this. Thanks Dhruba.

        Show
        Matei Zaharia added a comment - I committed this. Thanks Dhruba.
        Matei Zaharia made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Fix Version/s 0.21.0 [ 12313563 ]
        Resolution Fixed [ 1 ]
        Hide
        Hudson added a comment -

        Integrated in Hadoop-trunk #827 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/827/)
        . Allow setting the default value of maxRunningJobs for all
        pools. Contributed by Dhruba Borthakur.

        Show
        Hudson added a comment - Integrated in Hadoop-trunk #827 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/827/ ) . Allow setting the default value of maxRunningJobs for all pools. Contributed by Dhruba Borthakur.
        Owen O'Malley made changes -
        Component/s contrib/fair-share [ 12312456 ]
        Hide
        Robert Chansler added a comment -

        Editorial pass over all release notes prior to publication of 0.21.

        Show
        Robert Chansler added a comment - Editorial pass over all release notes prior to publication of 0.21.
        Robert Chansler made changes -
        Release Note Added a configuration parameter for setting a default limit on number of running jobs for all pools in the fair scheduler. New Fair Scheduler configuration parameter sets a default limit on number of running jobs for all pools.
        Tom White made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Patch Available Patch Available Open Open
        4d 22h 43m 1 dhruba borthakur 04/May/09 06:14
        Open Open Patch Available Patch Available
        2d 22h 18m 2 dhruba borthakur 04/May/09 06:15
        Patch Available Patch Available Resolved Resolved
        1d 47m 1 Matei Zaharia 05/May/09 07:03
        Resolved Resolved Closed Closed
        476d 14h 33m 1 Tom White 24/Aug/10 21:37

          People

          • Assignee:
            dhruba borthakur
            Reporter:
            dhruba borthakur
          • Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development