Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.2-alpha
    • Component/s: mrv2, scheduler
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress.

      1. MAPREDUCE-3451.v9.patch
        196 kB
        NO NAME
      2. MAPREDUCE-3451.v8.patch
        208 kB
        NO NAME
      3. MAPREDUCE-3451.v7.patch
        197 kB
        NO NAME
      4. MAPREDUCE-3451.v6.patch
        197 kB
        NO NAME
      5. MAPREDUCE-3451.v5.patch
        193 kB
        Harsh J
      6. MAPREDUCE-3451.v4.patch.txt
        193 kB
        NO NAME
      7. MAPREDUCE-3451.v3.patch.txt
        191 kB
        NO NAME
      8. MAPREDUCE-3451.v2.patch.txt
        176 kB
        NO NAME
      9. MAPREDUCE-3451.v1.patch.txt
        171 kB
        NO NAME

        Issue Links

        There are no Sub-Tasks for this issue.

          Activity

          Hide
          Harsh J added a comment -

          Hi Vitaly,

          The minResources (and also maxResources) are to be set as megabytes.

          Show
          Harsh J added a comment - Hi Vitaly, The minResources (and also maxResources) are to be set as megabytes.
          Hide
          Vitaly Kruglikov added a comment -

          NO NAME:
          FairScheduler.apt.vm description for the "minResources" queue property doesn't say what the units are: megabytes or bytes? For example, mapreduce.map.memory.mb is in megabytes. Could you please clarify the units for "minResources"? Many thanks!

          Show
          Vitaly Kruglikov added a comment - NO NAME : FairScheduler.apt.vm description for the "minResources" queue property doesn't say what the units are: megabytes or bytes? For example, mapreduce.map.memory.mb is in megabytes. Could you please clarify the units for "minResources"? Many thanks!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1136 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1136/)
          MAPREDUCE-3451. Amendment, excluding findbugs warnings (tucu) (Revision 1361436)

          Result = SUCCESS
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361436
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/dev-support/findbugs-exclude.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1136 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1136/ ) MAPREDUCE-3451 . Amendment, excluding findbugs warnings (tucu) (Revision 1361436) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361436 Files : /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/dev-support/findbugs-exclude.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1103 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1103/)
          MAPREDUCE-3451. Amendment, excluding findbugs warnings (tucu) (Revision 1361436)

          Result = FAILURE
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361436
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/dev-support/findbugs-exclude.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1103 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1103/ ) MAPREDUCE-3451 . Amendment, excluding findbugs warnings (tucu) (Revision 1361436) Result = FAILURE tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361436 Files : /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/dev-support/findbugs-exclude.xml
          Hide
          Ted Yu added a comment -

          MAPREDUCE-4445 is the correct JIRA, sorry about this.

          Show
          Ted Yu added a comment - MAPREDUCE-4445 is the correct JIRA, sorry about this.
          Hide
          Ted Yu added a comment -

          HBASE-6395 has been created for putting TestFSSchedulerApp in the right package.

          Show
          Ted Yu added a comment - HBASE-6395 has been created for putting TestFSSchedulerApp in the right package.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #2486 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2486/)
          MAPREDUCE-3451. Amendment, excluding findbugs warnings (tucu) (Revision 1361436)

          Result = FAILURE
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361436
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/dev-support/findbugs-exclude.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2486 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2486/ ) MAPREDUCE-3451 . Amendment, excluding findbugs warnings (tucu) (Revision 1361436) Result = FAILURE tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361436 Files : /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/dev-support/findbugs-exclude.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2532 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2532/)
          MAPREDUCE-3451. Amendment, excluding findbugs warnings (tucu) (Revision 1361436)

          Result = SUCCESS
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361436
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/dev-support/findbugs-exclude.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2532 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2532/ ) MAPREDUCE-3451 . Amendment, excluding findbugs warnings (tucu) (Revision 1361436) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361436 Files : /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/dev-support/findbugs-exclude.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2466 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2466/)
          MAPREDUCE-3451. Amendment, excluding findbugs warnings (tucu) (Revision 1361436)

          Result = SUCCESS
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361436
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/dev-support/findbugs-exclude.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2466 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2466/ ) MAPREDUCE-3451 . Amendment, excluding findbugs warnings (tucu) (Revision 1361436) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361436 Files : /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/dev-support/findbugs-exclude.xml
          Hide
          Alejandro Abdelnur added a comment -

          I'm on amending the patch

          Show
          Alejandro Abdelnur added a comment - I'm on amending the patch
          Hide
          Arun C Murthy added a comment -

          Patrick - the last comment/advice was to supress them (from Harsh). Why weren't they?

          Now, all patch builds are failing complaining about the findbugs warnings...

          Show
          Arun C Murthy added a comment - Patrick - the last comment/advice was to supress them (from Harsh). Why weren't they? Now, all patch builds are failing complaining about the findbugs warnings...
          Hide
          NO NAME added a comment -

          It's a one line change to the package header. The findbugs issues are discussed further up in this Jira and were also discussed during the last round of reviews.

          Show
          NO NAME added a comment - It's a one line change to the package header. The findbugs issues are discussed further up in this Jira and were also discussed during the last round of reviews.
          Hide
          NO NAME added a comment -

          Yes the error is related to the last minute movement of that test into the Fair package.

          Karthik - could you quickly patch this so we can get the build stabilized? If you can fix the findbugs quickly that would be great too - as i said earlier these are false positives but we might be able to coerce findbugs into not spouting warnings.

          Show
          NO NAME added a comment - Yes the error is related to the last minute movement of that test into the Fair package. Karthik - could you quickly patch this so we can get the build stabilized? If you can fix the findbugs quickly that would be great too - as i said earlier these are false positives but we might be able to coerce findbugs into not spouting warnings.
          Hide
          Karthik Kambatla added a comment -

          Do you suggest creating a clean-up JIRA and addressing these issues?

          Show
          Karthik Kambatla added a comment - Do you suggest creating a clean-up JIRA and addressing these issues?
          Hide
          Karthik Kambatla added a comment -

          Arun, in the last version of the patch, Patrick moved TestFSSchdulerApp from o.a.h.y.s.rm.scheduler to o.a.h.y.s.rm.scheduler.fair. He might have forgotten to update the package name accordingly. From what I understand, because of the wrong package name, the class file is still created one level above.

          Show
          Karthik Kambatla added a comment - Arun, in the last version of the patch, Patrick moved TestFSSchdulerApp from o.a.h.y.s.rm.scheduler to o.a.h.y.s.rm.scheduler.fair. He might have forgotten to update the package name accordingly. From what I understand, because of the wrong package name, the class file is still created one level above.
          Hide
          Arun C Murthy added a comment -

          Digging more - looks like I don't see the fair-scheduler tests show up in hadoop-yarn-server-resourcemanager-3.0.0-SNAPSHOT-tests.jar.

          Spoke too soon, I see the following:

          $ jar -tvf ./hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/target/hadoop-yarn-server-resourcemanager-3.0.0-SNAPSHOT-tests.jar | grep -i fair
               0 Fri Jul 13 10:37:20 PDT 2012 org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/
             324 Fri Jul 13 10:37:20 PDT 2012 org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler$1.class
            1463 Fri Jul 13 10:37:20 PDT 2012 org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler$MockClock.class
           25975 Fri Jul 13 10:37:20 PDT 2012 org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.class
          

          However, I don't see TestFSSchedulerApp, not sure why I don't see a compilation failure given the wrong package declaration.

          Patrick - any idea?

          Show
          Arun C Murthy added a comment - Digging more - looks like I don't see the fair-scheduler tests show up in hadoop-yarn-server-resourcemanager-3.0.0-SNAPSHOT-tests.jar. Spoke too soon, I see the following: $ jar -tvf ./hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/target/hadoop-yarn-server-resourcemanager-3.0.0-SNAPSHOT-tests.jar | grep -i fair 0 Fri Jul 13 10:37:20 PDT 2012 org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/ 324 Fri Jul 13 10:37:20 PDT 2012 org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler$1.class 1463 Fri Jul 13 10:37:20 PDT 2012 org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler$MockClock.class 25975 Fri Jul 13 10:37:20 PDT 2012 org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.class However, I don't see TestFSSchedulerApp, not sure why I don't see a compilation failure given the wrong package declaration. Patrick - any idea?
          Hide
          Arun C Murthy added a comment -

          Digging more - looks like I don't see the fair-scheduler tests show up in hadoop-yarn-server-resourcemanager-3.0.0-SNAPSHOT-tests.jar.

          Is that the reason why we don't see a compilation failure even though my IDE complains?

          Some maven work remaining, perhaps?

          Show
          Arun C Murthy added a comment - Digging more - looks like I don't see the fair-scheduler tests show up in hadoop-yarn-server-resourcemanager-3.0.0-SNAPSHOT-tests.jar. Is that the reason why we don't see a compilation failure even though my IDE complains? Some maven work remaining, perhaps?
          Hide
          Arun C Murthy added a comment -

          My IDE complains bitterly that TestFSSchdulerApp has the wrong package declaration.

          Also, my IDE complains that Queue needs an import.

          Show
          Arun C Murthy added a comment - My IDE complains bitterly that TestFSSchdulerApp has the wrong package declaration. Also, my IDE complains that Queue needs an import.
          Hide
          Arun C Murthy added a comment -

          I'm seeing another problem after I did a 'git pull' today after this commit.

          My IDE complains bitterly that TestFSSchdulerApp has the wrong package declaration.

          Alejandro Abdelnur: Should we revert this and fix the package problem and the findbugs warnings?

          Show
          Arun C Murthy added a comment - I'm seeing another problem after I did a 'git pull' today after this commit. My IDE complains bitterly that TestFSSchdulerApp has the wrong package declaration. Alejandro Abdelnur : Should we revert this and fix the package problem and the findbugs warnings?
          Hide
          Arun C Murthy added a comment -

          Unfortunately we've introduced findbugs warnings, Alejandro you missed them?

          I'll file another jira.

          Show
          Arun C Murthy added a comment - Unfortunately we've introduced findbugs warnings, Alejandro you missed them? I'll file another jira.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1135 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1135/)
          MAPREDUCE-3451. Port Fair Scheduler to MR2 (pwendell via tucu) (Revision 1361020)

          Result = SUCCESS
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361020
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/resource/Resources.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainer.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerEventType.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationConfigurationException.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AppSchedulable.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueue.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueueSchedulable.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSSchedulerApp.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerEventLog.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FifoAppComparator.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/NewJobWeightBooster.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/Schedulable.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingAlgorithms.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingMode.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/WeightAdjuster.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmController.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSSchedulerApp.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/FairScheduler.apt.vm
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1135 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1135/ ) MAPREDUCE-3451 . Port Fair Scheduler to MR2 (pwendell via tucu) (Revision 1361020) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361020 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/resource/Resources.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainer.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerEventType.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationConfigurationException.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AppSchedulable.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueue.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueueSchedulable.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSSchedulerApp.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerEventLog.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FifoAppComparator.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/NewJobWeightBooster.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/Schedulable.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingAlgorithms.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingMode.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/WeightAdjuster.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmController.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSSchedulerApp.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/FairScheduler.apt.vm
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1102 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1102/)
          MAPREDUCE-3451. Port Fair Scheduler to MR2 (pwendell via tucu) (Revision 1361020)

          Result = FAILURE
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361020
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/resource/Resources.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainer.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerEventType.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationConfigurationException.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AppSchedulable.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueue.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueueSchedulable.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSSchedulerApp.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerEventLog.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FifoAppComparator.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/NewJobWeightBooster.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/Schedulable.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingAlgorithms.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingMode.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/WeightAdjuster.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmController.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSSchedulerApp.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/FairScheduler.apt.vm
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1102 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1102/ ) MAPREDUCE-3451 . Port Fair Scheduler to MR2 (pwendell via tucu) (Revision 1361020) Result = FAILURE tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361020 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/resource/Resources.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainer.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerEventType.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationConfigurationException.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AppSchedulable.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueue.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueueSchedulable.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSSchedulerApp.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerEventLog.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FifoAppComparator.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/NewJobWeightBooster.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/Schedulable.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingAlgorithms.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingMode.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/WeightAdjuster.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmController.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSSchedulerApp.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/FairScheduler.apt.vm
          Hide
          NO NAME added a comment -

          We have several pieces of follow-up around this, and one of them can be looking at moving SchedulerApp to an interface (this would involve changing the other two schedulers as well to some extent).

          I'd like to see that change occur in tandem with better documentation for common scheduling components like SchedulerApp. The design goals of the RM scheduling code are currently opaque to anyone trying to add new code. Better documentation would make it clearer how to hook in a new scheduler the right way.

          Show
          NO NAME added a comment - We have several pieces of follow-up around this, and one of them can be looking at moving SchedulerApp to an interface (this would involve changing the other two schedulers as well to some extent). I'd like to see that change occur in tandem with better documentation for common scheduling components like SchedulerApp. The design goals of the RM scheduling code are currently opaque to anyone trying to add new code. Better documentation would make it clearer how to hook in a new scheduler the right way.
          Hide
          Arun C Murthy added a comment -

          As far as I can tell - the idea behind SchedulerApp is to deal with application lifecycle management from the perspective of the scheduler.

          I wish I had a chance to respond to Patrick before this was committed... IAC, we should plan to allow the schedulers to deviate in significant ways (which is the whole point of having multiple schedulers) and, as a result, minimizing interference is a key goal.

          Maybe we should open a jira to make SchedulerApp an interface?

          I originally kept it common across FifoScheduler and CapacityScheduler since, for the degenerate case of a single queue, they are identical (currently) and planned to pull it out as a separate one once they deviated.

          Show
          Arun C Murthy added a comment - As far as I can tell - the idea behind SchedulerApp is to deal with application lifecycle management from the perspective of the scheduler. I wish I had a chance to respond to Patrick before this was committed... IAC, we should plan to allow the schedulers to deviate in significant ways (which is the whole point of having multiple schedulers) and, as a result, minimizing interference is a key goal. Maybe we should open a jira to make SchedulerApp an interface? I originally kept it common across FifoScheduler and CapacityScheduler since, for the degenerate case of a single queue, they are identical (currently) and planned to pull it out as a separate one once they deviated.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #2479 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2479/)
          MAPREDUCE-3451. Port Fair Scheduler to MR2 (pwendell via tucu) (Revision 1361020)

          Result = FAILURE
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361020
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/resource/Resources.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainer.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerEventType.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationConfigurationException.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AppSchedulable.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueue.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueueSchedulable.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSSchedulerApp.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerEventLog.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FifoAppComparator.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/NewJobWeightBooster.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/Schedulable.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingAlgorithms.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingMode.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/WeightAdjuster.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmController.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSSchedulerApp.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/FairScheduler.apt.vm
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2479 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2479/ ) MAPREDUCE-3451 . Port Fair Scheduler to MR2 (pwendell via tucu) (Revision 1361020) Result = FAILURE tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361020 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/resource/Resources.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainer.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerEventType.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationConfigurationException.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AppSchedulable.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueue.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueueSchedulable.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSSchedulerApp.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerEventLog.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FifoAppComparator.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/NewJobWeightBooster.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/Schedulable.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingAlgorithms.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingMode.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/WeightAdjuster.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmController.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSSchedulerApp.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/FairScheduler.apt.vm
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2526 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2526/)
          MAPREDUCE-3451. Port Fair Scheduler to MR2 (pwendell via tucu) (Revision 1361020)

          Result = SUCCESS
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361020
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/resource/Resources.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainer.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerEventType.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationConfigurationException.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AppSchedulable.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueue.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueueSchedulable.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSSchedulerApp.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerEventLog.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FifoAppComparator.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/NewJobWeightBooster.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/Schedulable.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingAlgorithms.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingMode.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/WeightAdjuster.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmController.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSSchedulerApp.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/FairScheduler.apt.vm
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2526 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2526/ ) MAPREDUCE-3451 . Port Fair Scheduler to MR2 (pwendell via tucu) (Revision 1361020) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361020 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/resource/Resources.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainer.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerEventType.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationConfigurationException.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AppSchedulable.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueue.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueueSchedulable.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSSchedulerApp.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerEventLog.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FifoAppComparator.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/NewJobWeightBooster.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/Schedulable.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingAlgorithms.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingMode.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/WeightAdjuster.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmController.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSSchedulerApp.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/FairScheduler.apt.vm
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2460 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2460/)
          MAPREDUCE-3451. Port Fair Scheduler to MR2 (pwendell via tucu) (Revision 1361020)

          Result = SUCCESS
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361020
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/resource/Resources.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainer.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerEventType.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationConfigurationException.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AppSchedulable.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueue.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueueSchedulable.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSSchedulerApp.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerEventLog.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FifoAppComparator.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/NewJobWeightBooster.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/Schedulable.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingAlgorithms.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingMode.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/WeightAdjuster.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmController.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSSchedulerApp.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/FairScheduler.apt.vm
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2460 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2460/ ) MAPREDUCE-3451 . Port Fair Scheduler to MR2 (pwendell via tucu) (Revision 1361020) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361020 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/resource/Resources.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainer.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerEventType.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationConfigurationException.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AppSchedulable.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueue.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueueSchedulable.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSSchedulerApp.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerEventLog.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FifoAppComparator.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/NewJobWeightBooster.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/Schedulable.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingAlgorithms.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingMode.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/WeightAdjuster.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmController.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSSchedulerApp.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/FairScheduler.apt.vm
          Hide
          Alejandro Abdelnur added a comment -

          Nice job, thanks Patrick. Committed to trunk and branch-2.

          Show
          Alejandro Abdelnur added a comment - Nice job, thanks Patrick. Committed to trunk and branch-2.
          Hide
          Alejandro Abdelnur added a comment -

          +1

          Show
          Alejandro Abdelnur added a comment - +1
          Hide
          NO NAME added a comment -

          This patch moves a TestFSSchedulerApp into the fair/ test subdirectory.

          It also reverts whitespace fixes in SchedulerApp.java (so that file is now totally unchanged from trunk). That is out of scope for this JIRA.

          Show
          NO NAME added a comment - This patch moves a TestFSSchedulerApp into the fair/ test subdirectory. It also reverts whitespace fixes in SchedulerApp.java (so that file is now totally unchanged from trunk). That is out of scope for this JIRA.
          Hide
          NO NAME added a comment -

          This patch extends rather than modifies the SchedulerApp class for use in the Fair Scheduler.

          I cleaned up some tabs/whitespace issues in SchedulerApp.java and those are retained in this patch. But there are no code changes to SchedulerApp.java.

          Show
          NO NAME added a comment - This patch extends rather than modifies the SchedulerApp class for use in the Fair Scheduler. I cleaned up some tabs/whitespace issues in SchedulerApp.java and those are retained in this patch. But there are no code changes to SchedulerApp.java.
          Hide
          NO NAME added a comment -

          Hey Arun - I think extending it is a better move for now.

          If we decided to move SchedulerApp to an interface, 95% of the implementation would be the same between the FS/CS. The only differences are in the way delay scheduling works for the FairScheduler.

          As far as I can tell - the idea behind SchedulerApp is to deal with application lifecycle management from the perspective of the scheduler. That's helpful to have and made writing the FS simpler. This is likely to be useful for future schedulers well. In terms of maximizing code sharing I would be in favor of keeping it collectively managed.

          Show
          NO NAME added a comment - Hey Arun - I think extending it is a better move for now. If we decided to move SchedulerApp to an interface, 95% of the implementation would be the same between the FS/CS. The only differences are in the way delay scheduling works for the FairScheduler. As far as I can tell - the idea behind SchedulerApp is to deal with application lifecycle management from the perspective of the scheduler. That's helpful to have and made writing the FS simpler. This is likely to be useful for future schedulers well. In terms of maximizing code sharing I would be in favor of keeping it collectively managed.
          Hide
          Arun C Murthy added a comment -

          In fact, we should probably make SchedulerApp etc. an interface and have different implementations for CS, FS etc. to minimize interference. Thoughts?

          Show
          Arun C Murthy added a comment - In fact, we should probably make SchedulerApp etc. an interface and have different implementations for CS, FS etc. to minimize interference. Thoughts?
          Hide
          Arun C Murthy added a comment -

          Patrick I took a brief look, mostly looks great! I agree having different schedulers is very useful.

          Minor nits: I don't think it's worth modifying common infrastructure such as SchedulerApp right now - maybe you can just extend them where necessary?

          Show
          Arun C Murthy added a comment - Patrick I took a brief look, mostly looks great! I agree having different schedulers is very useful. Minor nits: I don't think it's worth modifying common infrastructure such as SchedulerApp right now - maybe you can just extend them where necessary?
          Hide
          Tom White added a comment -

          Thanks for summarizing the goals here Patrick. I agree that having a single scheduler is not realistic at this point.

          The latest change (and test) supporting the "pool" element in the allocation XML file for backwards compatibility looks good to me.

          +1 (pending Jenkins)

          Show
          Tom White added a comment - Thanks for summarizing the goals here Patrick. I agree that having a single scheduler is not realistic at this point. The latest change (and test) supporting the "pool" element in the allocation XML file for backwards compatibility looks good to me. +1 (pending Jenkins)
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12536147/MAPREDUCE-3451.v7.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 9 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2576//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2576//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2576//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12536147/MAPREDUCE-3451.v7.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 9 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2576//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2576//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2576//console This message is automatically generated.
          Hide
          NO NAME added a comment -

          So I spent some time offline talking with Tom and I think we're ready to move forward with committing this to MR2. The primary goals are:

          1) Let people who use the MR1 fair scheduler have a drop in equivalent for MR2

          • and -
            2) Keep as much code shared as possible between MR2 schedulers given that the overlap in substantial functionality

          I think we're clearly accomplishing (1) and well on our way to (2), but subsequent patches might find more opportunity for code re-use.

          This also doesn't include some necessities like the Web UI and some nice-to-have extensions like hierarchical queues. That said I'd like to get this committed as a starting point build from there, given that it's already a pretty large commit.

          Show
          NO NAME added a comment - So I spent some time offline talking with Tom and I think we're ready to move forward with committing this to MR2. The primary goals are: 1) Let people who use the MR1 fair scheduler have a drop in equivalent for MR2 and - 2) Keep as much code shared as possible between MR2 schedulers given that the overlap in substantial functionality I think we're clearly accomplishing (1) and well on our way to (2), but subsequent patches might find more opportunity for code re-use. This also doesn't include some necessities like the Web UI and some nice-to-have extensions like hierarchical queues. That said I'd like to get this committed as a starting point build from there, given that it's already a pretty large commit.
          Hide
          NO NAME added a comment -

          Forgot to include some whitespace fixes in last patch

          Show
          NO NAME added a comment - Forgot to include some whitespace fixes in last patch
          Hide
          NO NAME added a comment -

          Bringing this back to trunk again and adding one small change to make config files more backwards incompatible.

          Show
          NO NAME added a comment - Bringing this back to trunk again and adding one small change to make config files more backwards incompatible.
          Hide
          NO NAME added a comment -

          I'm not sure I understand what it would mean to have a single featureful scheduler.

          There is already an abstraction layer built into YARN to allow multiple scheudling policies. Aside from the logic specific to the policy itself, the code for dealing with resource management is re-used amongst schedulers. That is, when someone is using the Fair Scheduler (vs the Capacity or FIFO scheduler) they are using mostly the same RM code save for the scheduling logic itself. Furthermore, several shared classes are used by both the Fair and Capacity schedulers, such as the Queue interface and the SchedulerApp class - these common classes are "tested" by users of both schedulers.

          The alternative suggested seems to be having a monolithic scheduler class that can enact different policies depending on some configuration option. Any sensible implementation of that approach would abstract out the scheduling policy and I think you'd get what's already there now.

          Or maybe the idea is that everyone using Hadoop will want the same scheduling policy (modulo minor configurations like timeouts, capacities, etc.). That seems unlikely to me given the diversity of Hadoop deployments and the fact that both of the currently available schedulers in MR1 are widely used.

          Also, just a note - adding preemption to the capacity scheduler will not make it equal the Fair Scheduler. These are fundamentally different approaches which will become substantailly more different when multiple resource types are added to the YARN RM stack.

          I strongly agree that having single implementations of common classes is a win in terms of support and testability - but I think this implementation mostly acheives that goal.

          Show
          NO NAME added a comment - I'm not sure I understand what it would mean to have a single featureful scheduler. There is already an abstraction layer built into YARN to allow multiple scheudling policies. Aside from the logic specific to the policy itself, the code for dealing with resource management is re-used amongst schedulers. That is, when someone is using the Fair Scheduler (vs the Capacity or FIFO scheduler) they are using mostly the same RM code save for the scheduling logic itself. Furthermore, several shared classes are used by both the Fair and Capacity schedulers, such as the Queue interface and the SchedulerApp class - these common classes are "tested" by users of both schedulers. The alternative suggested seems to be having a monolithic scheduler class that can enact different policies depending on some configuration option. Any sensible implementation of that approach would abstract out the scheduling policy and I think you'd get what's already there now. Or maybe the idea is that everyone using Hadoop will want the same scheduling policy (modulo minor configurations like timeouts, capacities, etc.). That seems unlikely to me given the diversity of Hadoop deployments and the fact that both of the currently available schedulers in MR1 are widely used. Also, just a note - adding preemption to the capacity scheduler will not make it equal the Fair Scheduler. These are fundamentally different approaches which will become substantailly more different when multiple resource types are added to the YARN RM stack. I strongly agree that having single implementations of common classes is a win in terms of support and testability - but I think this implementation mostly acheives that goal.
          Hide
          Matei Zaharia added a comment -

          Having a single very feature-rich scheduler is certainly useful, but that's a separate discussion. My point was just that some people want compatibility with the MR1 Fair Scheduler as a feature – not having that is an obstacle to their adopting MR2.

          I also think it would be hard to design a single scheduler that will support all the features people might want in the future, for the same reason that it was hard to get different development groups to build a single scheduler in MR1. You just can't anticipate all the needs, and how various features will interact – for example, Yahoo! had to remove preemption from the capacity scheduler because it complicated the design. We are also planning to add some features to the Fair Scheduler ourselves, such as multi-resource fairness, that might require major changes to other schedulers even if we were to wait on them adopting the existing features.

          Show
          Matei Zaharia added a comment - Having a single very feature-rich scheduler is certainly useful, but that's a separate discussion. My point was just that some people want compatibility with the MR1 Fair Scheduler as a feature – not having that is an obstacle to their adopting MR2. I also think it would be hard to design a single scheduler that will support all the features people might want in the future, for the same reason that it was hard to get different development groups to build a single scheduler in MR1. You just can't anticipate all the needs, and how various features will interact – for example, Yahoo! had to remove preemption from the capacity scheduler because it complicated the design. We are also planning to add some features to the Fair Scheduler ourselves, such as multi-resource fairness, that might require major changes to other schedulers even if we were to wait on them adopting the existing features.
          Hide
          Tom White added a comment -

          Matei - I'm not against this going in (although we should put it in a separate module), but I thought it would be a good time to have a discussion about supporting a single main scheduler in YARN. If we could combine features from the various schedulers in MR1 to YARN then we would have less code to support and test, and it would be less confusing for users. What do you think? Do you agree that this would be a useful goal?

          Show
          Tom White added a comment - Matei - I'm not against this going in (although we should put it in a separate module), but I thought it would be a good time to have a discussion about supporting a single main scheduler in YARN. If we could combine features from the various schedulers in MR1 to YARN then we would have less code to support and test, and it would be less confusing for users. What do you think? Do you agree that this would be a useful goal?
          Hide
          Matei Zaharia added a comment -

          I think the fair scheduler can be one of the alternative schedulers. Why not leave it in? Apart from the features it implements, another major advantage is that it will be compatible with fair scheduler config files that people have set up before moving to MR2. This was one of the main reasons why people have been asking us for it – they want to be sure that their cluster will share resources the same way as before.

          Show
          Matei Zaharia added a comment - I think the fair scheduler can be one of the alternative schedulers. Why not leave it in? Apart from the features it implements, another major advantage is that it will be compatible with fair scheduler config files that people have set up before moving to MR2. This was one of the main reasons why people have been asking us for it – they want to be sure that their cluster will share resources the same way as before.
          Hide
          Tom White added a comment -

          Rather than simply porting the Fair Scheduler to MR2, I think we should consider whether the Capacity Scheduler can support the majority of use cases. Having a single scheduler would simplify things for users and administrators. Just like we merged the features of the MapReduce MultipleOutputs and MultipleOutputFormat libraries, moving to a single main scheduler would reduce confusion.

          Capacity Scheduler lacks a few features that FS has - but there are JIRAs for these (see MAPREDUCE-3210, MAPREDUCE-3938, MAPREDUCE-4257). Also, note that this doesn't preclude having other schedulers, since they are pluggable in MR2, so having other schedulers for experimentation, other use cases etc is still supported.

          Show
          Tom White added a comment - Rather than simply porting the Fair Scheduler to MR2, I think we should consider whether the Capacity Scheduler can support the majority of use cases. Having a single scheduler would simplify things for users and administrators. Just like we merged the features of the MapReduce MultipleOutputs and MultipleOutputFormat libraries, moving to a single main scheduler would reduce confusion. Capacity Scheduler lacks a few features that FS has - but there are JIRAs for these (see MAPREDUCE-3210 , MAPREDUCE-3938 , MAPREDUCE-4257 ). Also, note that this doesn't preclude having other schedulers, since they are pluggable in MR2, so having other schedulers for experimentation, other use cases etc is still supported.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12526140/MAPREDUCE-3451.v5.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 9 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2371//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2371//artifact/trunk/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2371//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12526140/MAPREDUCE-3451.v5.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 9 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2371//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2371//artifact/trunk/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2371//console This message is automatically generated.
          Hide
          Harsh J added a comment -

          Per MAPREDUCE-3812, I've tweaked your patch a bit to make it use the new configs for min/max RM allocation (memory for containers).

          One thing I noticed when tweaking your patch, is that there are quite a bit of trailing white-spaces and possibly a few occurrences of tab+space mix you may wanna clean up.

          The diff I've added to your patch overall is below:

          diff --git a/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java b/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
          index 35eafc5..e888be8 100644
          --- a/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
          +++ b/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
          @@ -4,6 +4,7 @@ import java.io.File;
           
           import org.apache.hadoop.conf.Configuration;
           import org.apache.hadoop.yarn.api.records.Resource;
          +import org.apache.hadoop.yarn.conf.YarnConfiguration;
           import org.apache.hadoop.yarn.server.resourcemanager.resource.Resources;
           
           public class FairSchedulerConfiguration extends Configuration {
          @@ -13,15 +14,7 @@ public class FairSchedulerConfiguration extends Configuration {
             
             protected static final String ALLOCATION_FILE = CONF_PREFIX + "allocation.file";
             protected static final String EVENT_LOG_DIR = "eventlog.dir";
          -  
          -  /** The minimum size container the scheduler will ever allocate. */
          -  protected static final String MINIMUM_MEMORY_MB = CONF_PREFIX + "minimum-allocation-mb";
          -  protected static final int    DEFAULT_MINIMUM_MEMORY_MB = 512;
          -  
          -  /** The maximum size container the scheduler will ever allocate. */
          -  protected static final String MAXIMUM_ALLOCATION_MB = CONF_PREFIX + "maximum-allocation-mb";
          -  protected static final int    DEFAULT_MAXIMUM_ALLOCATION_MB = 10240;
          -  
          +
             /** Whether to use the user name as the queue name (instead of "default") if
              * the request does not specify a queue. */
             protected static final String  USER_AS_DEFAULT_QUEUE = CONF_PREFIX + "user-as-default-queue";
          @@ -60,17 +53,21 @@ public class FairSchedulerConfiguration extends Configuration {
               super(conf);
               addResource(FS_CONFIGURATION_FILE);
             }
          -    
          +
             public Resource getMinimumMemoryAllocation() {
          -    int mem = getInt(MINIMUM_MEMORY_MB, DEFAULT_MINIMUM_MEMORY_MB);
          +    int mem = getInt(
          +        YarnConfiguration.RM_SCHEDULER_MINIMUM_ALLOCATION_MB,
          +        YarnConfiguration.DEFAULT_RM_SCHEDULER_MINIMUM_ALLOCATION_MB);
               return Resources.createResource(mem);
             }
          -  
          +
             public Resource getMaximumMemoryAllocation() {
          -    int mem = getInt(MAXIMUM_ALLOCATION_MB, DEFAULT_MAXIMUM_ALLOCATION_MB);
          +    int mem = getInt(
          +        YarnConfiguration.RM_SCHEDULER_MAXIMUM_ALLOCATION_MB,
          +        YarnConfiguration.DEFAULT_RM_SCHEDULER_MAXIMUM_ALLOCATION_MB);
               return Resources.createResource(mem);
             }
          -  
          +
             public boolean getUserAsDefaultQueue() {
               return getBoolean(USER_AS_DEFAULT_QUEUE, DEFAULT_USER_AS_DEFAULT_QUEUE);
             }
          

          Okay looks like that patch fixed everything except for the findbugs, there are some member variables which are written only while holding a lock but can be read outside of the lock and findbugs doesn't like this. I consider these false positives - any idea how to suppress?

          I didn't take a look at which specific mem-var you're discussing here, but what happens if one reads it when its being updated? Does your locking mech cover this scenario too? I remember making the same mistake a while ago at https://issues.apache.org/jira/browse/MAPREDUCE-1347?focusedCommentId=13003257&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13003257 but that may be irrelevant here.

          In case you wish to add it to ignores, see file hadoop-mapreduce-project/hadoop-yarn/dev-support/findbugs-exclude.xml and add appropriate entries for your discovered warnings to suppress them. I'd recommend suppressing them after the review/over another ticket and just leave as a comment for now that this may not be harmful as it looks.

          Show
          Harsh J added a comment - Per MAPREDUCE-3812 , I've tweaked your patch a bit to make it use the new configs for min/max RM allocation (memory for containers). One thing I noticed when tweaking your patch, is that there are quite a bit of trailing white-spaces and possibly a few occurrences of tab+space mix you may wanna clean up. The diff I've added to your patch overall is below: diff --git a/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java b/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java index 35eafc5..e888be8 100644 --- a/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java +++ b/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java @@ -4,6 +4,7 @@ import java.io.File; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.yarn.api.records.Resource; + import org.apache.hadoop.yarn.conf.YarnConfiguration; import org.apache.hadoop.yarn.server.resourcemanager.resource.Resources; public class FairSchedulerConfiguration extends Configuration { @@ -13,15 +14,7 @@ public class FairSchedulerConfiguration extends Configuration { protected static final String ALLOCATION_FILE = CONF_PREFIX + "allocation.file" ; protected static final String EVENT_LOG_DIR = "eventlog.dir" ; - - /** The minimum size container the scheduler will ever allocate. */ - protected static final String MINIMUM_MEMORY_MB = CONF_PREFIX + "minimum-allocation-mb" ; - protected static final int DEFAULT_MINIMUM_MEMORY_MB = 512; - - /** The maximum size container the scheduler will ever allocate. */ - protected static final String MAXIMUM_ALLOCATION_MB = CONF_PREFIX + "maximum-allocation-mb" ; - protected static final int DEFAULT_MAXIMUM_ALLOCATION_MB = 10240; - + /** Whether to use the user name as the queue name (instead of " default " ) if * the request does not specify a queue. */ protected static final String USER_AS_DEFAULT_QUEUE = CONF_PREFIX + "user-as- default -queue" ; @@ -60,17 +53,21 @@ public class FairSchedulerConfiguration extends Configuration { super (conf); addResource(FS_CONFIGURATION_FILE); } - + public Resource getMinimumMemoryAllocation() { - int mem = getInt(MINIMUM_MEMORY_MB, DEFAULT_MINIMUM_MEMORY_MB); + int mem = getInt( + YarnConfiguration.RM_SCHEDULER_MINIMUM_ALLOCATION_MB, + YarnConfiguration.DEFAULT_RM_SCHEDULER_MINIMUM_ALLOCATION_MB); return Resources.createResource(mem); } - + public Resource getMaximumMemoryAllocation() { - int mem = getInt(MAXIMUM_ALLOCATION_MB, DEFAULT_MAXIMUM_ALLOCATION_MB); + int mem = getInt( + YarnConfiguration.RM_SCHEDULER_MAXIMUM_ALLOCATION_MB, + YarnConfiguration.DEFAULT_RM_SCHEDULER_MAXIMUM_ALLOCATION_MB); return Resources.createResource(mem); } - + public boolean getUserAsDefaultQueue() { return getBoolean(USER_AS_DEFAULT_QUEUE, DEFAULT_USER_AS_DEFAULT_QUEUE); } Okay looks like that patch fixed everything except for the findbugs, there are some member variables which are written only while holding a lock but can be read outside of the lock and findbugs doesn't like this. I consider these false positives - any idea how to suppress? I didn't take a look at which specific mem-var you're discussing here, but what happens if one reads it when its being updated? Does your locking mech cover this scenario too? I remember making the same mistake a while ago at https://issues.apache.org/jira/browse/MAPREDUCE-1347?focusedCommentId=13003257&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13003257 but that may be irrelevant here. In case you wish to add it to ignores, see file hadoop-mapreduce-project/hadoop-yarn/dev-support/findbugs-exclude.xml and add appropriate entries for your discovered warnings to suppress them. I'd recommend suppressing them after the review/over another ticket and just leave as a comment for now that this may not be harmful as it looks.
          Hide
          NO NAME added a comment -

          Okay looks like that patch fixed everything except for the findbugs, there are some member variables which are written only while holding a lock but can be read outside of the lock and findbugs doesn't like this. I consider these false positives - any idea how to suppress?

          Show
          NO NAME added a comment - Okay looks like that patch fixed everything except for the findbugs, there are some member variables which are written only while holding a lock but can be read outside of the lock and findbugs doesn't like this. I consider these false positives - any idea how to suppress?
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12526086/MAPREDUCE-3451.v4.patch.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 9 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2369//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2369//artifact/trunk/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2369//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12526086/MAPREDUCE-3451.v4.patch.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 9 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2369//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2369//artifact/trunk/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2369//console This message is automatically generated.
          Hide
          NO NAME added a comment -

          Just added a patch addressing the legitimate findbugs and javadoc warnings. There were some findbugs warnings about concurrency that seemed like false positives - not sure if there is a way to suppress them.

          I can't re-create the core test failures locally... they seem related to port contention issues and Kerberos issues around the hostname of the Hudson machine. I don't see any direct relationship between these test failures and the code in this patch. Could we at least re-run this in Hudson to make sure that these tests are deterministically failing?

          Show
          NO NAME added a comment - Just added a patch addressing the legitimate findbugs and javadoc warnings. There were some findbugs warnings about concurrency that seemed like false positives - not sure if there is a way to suppress them. I can't re-create the core test failures locally... they seem related to port contention issues and Kerberos issues around the hostname of the Hudson machine. I don't see any direct relationship between these test failures and the code in this patch. Could we at least re-run this in Hudson to make sure that these tests are deterministically failing?
          Hide
          Luke Lu added a comment -

          It seems like it’s not easy to trigger a 404 from within a view, rather than at the URL router.

          You can trigger a 404 or any http response code anywhere (preferably in controller code as view is not guaranteed to be fully buffered for long responses by design) by calling Controller#setStatus (via context().setStatus in views (if it's not yet flushed) and then throw an WebAppException (do not use generic RuntimeException). You can bind ErrorPage.class in a webapp to any view class if you want a custom error page.

          Show
          Luke Lu added a comment - It seems like it’s not easy to trigger a 404 from within a view, rather than at the URL router. You can trigger a 404 or any http response code anywhere (preferably in controller code as view is not guaranteed to be fully buffered for long responses by design) by calling Controller#setStatus (via context().setStatus in views (if it's not yet flushed) and then throw an WebAppException (do not use generic RuntimeException). You can bind ErrorPage.class in a webapp to any view class if you want a custom error page.
          Hide
          Ahmed Radwan added a comment -

          Thanks Patrick for the comments, +1

          Show
          Ahmed Radwan added a comment - Thanks Patrick for the comments, +1
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12522923/MAPREDUCE-3451.v3.patch.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          -1 javadoc. The javadoc tool appears to have generated 2 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 21 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.yarn.server.TestContainerManagerSecurity
          org.apache.hadoop.yarn.server.resourcemanager.security.TestApplicationTokens
          org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService
          org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry
          org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization
          org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs
          org.apache.hadoop.mapred.TestClientRedirect
          org.apache.hadoop.mapreduce.security.TestJHSSecurity

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2259//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2259//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2259//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12522923/MAPREDUCE-3451.v3.patch.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified test files. -1 javadoc. The javadoc tool appears to have generated 2 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 21 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.server.TestContainerManagerSecurity org.apache.hadoop.yarn.server.resourcemanager.security.TestApplicationTokens org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs org.apache.hadoop.mapred.TestClientRedirect org.apache.hadoop.mapreduce.security.TestJHSSecurity +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2259//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2259//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2259//console This message is automatically generated.
          Hide
          Tom White added a comment -

          Patrick, thanks for addressing the feedback and also for adding documentation. I think it's fine to improve test coverage and revisit the configuration format in follow on JIRAs, so I'm +1 for committing this. I'm going to run this through Jenkins now.

          Show
          Tom White added a comment - Patrick, thanks for addressing the feedback and also for adding documentation. I think it's fine to improve test coverage and revisit the configuration format in follow on JIRAs, so I'm +1 for committing this. I'm going to run this through Jenkins now.
          Hide
          NO NAME added a comment -

          New patch including response to feedback and documentation file.

          Show
          NO NAME added a comment - New patch including response to feedback and documentation file.
          Hide
          NO NAME added a comment -

          Thanks for the feedback. About to attach a patch with the following changes:

          • Updated the terminology from “pool” to “queue” to be consistent.
          • I added annotations to all classes.
          • Fixed the debug → log statement
          • I modified the web UI to give a more precise error message in the logs, but it still has a generic 500 in the web page. MRv2 uses a custom built MVC framework based on Guice dependency injection. I’m embarrassed to say that I spend 40 minutes trying to figure out how to make it render a 404 and couldn’t. It seems like it’s not easy to trigger a 404 from within a view, rather than at the URL router.
          • Extra license header removed
          • Noted spelling errors fixed.
          • There is extensive test coverage in the MRS FS I’d like to get there eventually but probably not with this first commit. I tried to included tests of most of the core functionality. We can add this as a JIRA, though.
          • I consolidated the configuration constants as you suggested. For now I’m going to leave the queue manifest file as its own thing unless this is a deal-breaker. There is a lot of logic in parsing that file which I’d prefer to leave as-is since it’s very well tested. While the term names have changed it’s essentially identical configuration to the MR1 FS, so leaving things as-is should ease the transition for people running current clusters. It’s also a fundamentally hierarchical type of configuration, and the apache conf is all flat key/value space which will be much more clunky.

          I haven’t done much testing other than on a single node. One reason I want to get this in is that some are waiting on having something in trunk to start testing. The build is stable and the unit tests pass, so I hope that is sufficient to get this in there with the caveat that things may be brittle when run on a real cluster.

          I also added an updated version of the documentation (even the MR1 version wasn’t totally up-to-date with changes that have been made in the configuration options). This will need some more work going forward but it should give people to the basic configuration/installation info they need to get started testing this.

          Show
          NO NAME added a comment - Thanks for the feedback. About to attach a patch with the following changes: Updated the terminology from “pool” to “queue” to be consistent. I added annotations to all classes. Fixed the debug → log statement I modified the web UI to give a more precise error message in the logs, but it still has a generic 500 in the web page. MRv2 uses a custom built MVC framework based on Guice dependency injection. I’m embarrassed to say that I spend 40 minutes trying to figure out how to make it render a 404 and couldn’t. It seems like it’s not easy to trigger a 404 from within a view, rather than at the URL router. Extra license header removed Noted spelling errors fixed. There is extensive test coverage in the MRS FS I’d like to get there eventually but probably not with this first commit. I tried to included tests of most of the core functionality. We can add this as a JIRA, though. I consolidated the configuration constants as you suggested. For now I’m going to leave the queue manifest file as its own thing unless this is a deal-breaker. There is a lot of logic in parsing that file which I’d prefer to leave as-is since it’s very well tested. While the term names have changed it’s essentially identical configuration to the MR1 FS, so leaving things as-is should ease the transition for people running current clusters. It’s also a fundamentally hierarchical type of configuration, and the apache conf is all flat key/value space which will be much more clunky. I haven’t done much testing other than on a single node. One reason I want to get this in is that some are waiting on having something in trunk to start testing. The build is stable and the unit tests pass, so I hope that is sufficient to get this in there with the caveat that things may be brittle when run on a real cluster. I also added an updated version of the documentation (even the MR1 version wasn’t totally up-to-date with changes that have been made in the configuration options). This will need some more work going forward but it should give people to the basic configuration/installation info they need to get started testing this.
          Hide
          Aaron T. Myers added a comment -

          For the documentation pointer (hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/) - I'm unfamiliar with the markup being used in these documents (it looks like the Velocity engine maybe?).

          Take a look at this page for info on the markup: http://maven.apache.org/doxia/references/apt-format.html

          Is there any way to render these files locally as I'm writing some preliminary docs for the FS?

          From the repo root, `cd hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site' and run `mvn site'. You can then look at the rendered page by browsing to "file:///<path to repo>/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/target/site/<file name>.html"

          Show
          Aaron T. Myers added a comment - For the documentation pointer (hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/) - I'm unfamiliar with the markup being used in these documents (it looks like the Velocity engine maybe?). Take a look at this page for info on the markup: http://maven.apache.org/doxia/references/apt-format.html Is there any way to render these files locally as I'm writing some preliminary docs for the FS? From the repo root, `cd hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site' and run `mvn site'. You can then look at the rendered page by browsing to "file:///<path to repo>/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/target/site/<file name>.html"
          Hide
          NO NAME added a comment -

          Hey Tom/Ahmed,

          Thanks for the comments - I'm addressing them now. For the documentation pointer (hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/) - I'm unfamiliar with the markup being used in these documents (it looks like the Velocity engine maybe?). Is there any way to render these files locally as I'm writing some preliminary docs for the FS?

          • Patrick
          Show
          NO NAME added a comment - Hey Tom/Ahmed, Thanks for the comments - I'm addressing them now. For the documentation pointer (hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/) - I'm unfamiliar with the markup being used in these documents (it looks like the Velocity engine maybe?). Is there any way to render these files locally as I'm writing some preliminary docs for the FS? Patrick
          Hide
          Ahmed Radwan added a comment -

          Hi Patrick,

          There are also these comments/nits:

          • RMContainer.java: Added a unused "import org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl;"
          • Many of the inline docs have spelling mistakes. Please review them.
          • SchedulerApp.java: The following was removed from the docs: [Each time the scheduler
            asks the application for a task at this priority, it is incremented,
            and each time the application successfully schedules a task, it
            is reset to 0.], please add it back.
          • Add javadocs and setup and configuration docs.
          Show
          Ahmed Radwan added a comment - Hi Patrick, There are also these comments/nits: RMContainer.java: Added a unused "import org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl;" Many of the inline docs have spelling mistakes. Please review them. SchedulerApp.java: The following was removed from the docs: [Each time the scheduler asks the application for a task at this priority, it is incremented, and each time the application successfully schedules a task, it is reset to 0.], please add it back. Add javadocs and setup and configuration docs.
          Hide
          Tom White added a comment -

          Overall this looks like a great addition. Initial review feedback:

          • It looks like what were called "pools" in the MR1 version are now "queues", to fit with the term that the capacity scheduler uses. Is that correct? If so, then the fair scheduler code should use the same term throughout.
          • What's the locking order for the fair scheduler classes and the classes that call them? It might be worth documenting it somewhere in the code.
          • I noticed that not all the unit tests from MR1 have been ported. Are you planning on including the rest?
          • Configuration strings are duplicated and spread through different classes. How about creating a FairSchedulerConfiguration class to hold all the constants (like capacity scheduler does)?
          • Seeing that the configuration is changing compared to MR1, is it worth changing the allocations file to be in Hadoop configuration format? Having one less format to deal with would be an improvement IMO.
          • Annotate all classes as @Private @Unstable. Some are not annotated at all, even though they should be considered private.
          • PoolSchedulable.assignContainer has a log line which is called frequently ("Node offered to pool"), so should really be debug level.
          • The Scheduler UI link gives a 500 error - I know you plan on fixing this, but I would have expected a 404.
          • Nit: PoolManager has two license headers
          • Nit: Resources class spelling: "Mutliply"
          • Nit: SchedulerApp - schedulingOpportunities comment has been changed from javadoc comment in MR1 - change back?

          What testing have you done? I tried it on a pseudo distributed cluster and successfully ran a job with the fair scheduler.

          This will need some documentation (in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/) - are you planning on doing that as a separate JIRA? It would also be useful to call out the configuration differences compared to the MR1 fair scheduler. (E.g. I noticed that yarn.scheduler.fair.user-as-default-queue is new.)

          Show
          Tom White added a comment - Overall this looks like a great addition. Initial review feedback: It looks like what were called "pools" in the MR1 version are now "queues", to fit with the term that the capacity scheduler uses. Is that correct? If so, then the fair scheduler code should use the same term throughout. What's the locking order for the fair scheduler classes and the classes that call them? It might be worth documenting it somewhere in the code. I noticed that not all the unit tests from MR1 have been ported. Are you planning on including the rest? Configuration strings are duplicated and spread through different classes. How about creating a FairSchedulerConfiguration class to hold all the constants (like capacity scheduler does)? Seeing that the configuration is changing compared to MR1, is it worth changing the allocations file to be in Hadoop configuration format? Having one less format to deal with would be an improvement IMO. Annotate all classes as @Private @Unstable. Some are not annotated at all, even though they should be considered private. PoolSchedulable.assignContainer has a log line which is called frequently ("Node offered to pool"), so should really be debug level. The Scheduler UI link gives a 500 error - I know you plan on fixing this, but I would have expected a 404. Nit: PoolManager has two license headers Nit: Resources class spelling: "Mutliply" Nit: SchedulerApp - schedulingOpportunities comment has been changed from javadoc comment in MR1 - change back? What testing have you done? I tried it on a pseudo distributed cluster and successfully ran a job with the fair scheduler. This will need some documentation (in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/) - are you planning on doing that as a separate JIRA? It would also be useful to call out the configuration differences compared to the MR1 fair scheduler. (E.g. I noticed that yarn.scheduler.fair.user-as-default-queue is new.)
          Hide
          NO NAME added a comment -

          Hey Ahmed,

          MR2 does support multiple resource types, but as you point out, currently only memory is implemented. Right now this scheduler basically treats memory similarly to the way slots were handled in the old fair scheduler, with the only key difference being that memory can be allocated in variable size chunks.

          There is no widely accepted definition of what "fairness" means with multiple resource types, but there have been proposals from the academic community (see http://static.usenix.org/events/nsdi11/tech/full_papers/Ghodsi.pdf) which could ultimately be factored into the scheduler if and when multiple resource types are introduced. Having several resource dimensions complicates the scheduling decision substantially and will require major changes to existing schedulers (such as the capacity scheduler) as well. In all of these cases, existing approaches depend on a single fungible resource about which you can express minimum capacities (or shares), target capacities (or shares), etc. These semantics don't make sense with multiple resource types.

          Right now the preemption logic preempts lowest priority containers first. This is the best general-purpose metric I could think of for choosing what to preempt.

          I will include a write-up about deployment in the scheduler's current form as well in an upcoming patch.

          Show
          NO NAME added a comment - Hey Ahmed, MR2 does support multiple resource types, but as you point out, currently only memory is implemented. Right now this scheduler basically treats memory similarly to the way slots were handled in the old fair scheduler, with the only key difference being that memory can be allocated in variable size chunks. There is no widely accepted definition of what "fairness" means with multiple resource types, but there have been proposals from the academic community (see http://static.usenix.org/events/nsdi11/tech/full_papers/Ghodsi.pdf ) which could ultimately be factored into the scheduler if and when multiple resource types are introduced. Having several resource dimensions complicates the scheduling decision substantially and will require major changes to existing schedulers (such as the capacity scheduler) as well. In all of these cases, existing approaches depend on a single fungible resource about which you can express minimum capacities (or shares), target capacities (or shares), etc. These semantics don't make sense with multiple resource types. Right now the preemption logic preempts lowest priority containers first. This is the best general-purpose metric I could think of for choosing what to preempt. I will include a write-up about deployment in the scheduler's current form as well in an upcoming patch.
          Hide
          Ahmed Radwan added a comment -

          Thanks Patrick.

          The patch successfully applies and compiles on trunk. The included new tests also successfully run.

          I'll postpone specific code comments, since I have some basic questions:

          The old fair scheduler used slots as the basic allocation units, and calculating shares and other features (like preemption) relied on this model. In MR2 we have containers with variable resource attributes. Currently the only supported attribute is memory, but it is planned to add more attributes (network bandwidth, cpu, etc.). So it is not clear for me how this mapping is handled. Are you envisioning combing such attributes for calculating shares, how is this done? and how preemption will be done in such case, for example in the old MR1, the most recent slot was killed. But now slots are not equal and this complicate the choices. I am just using preemption as an example. Can you please elaborate on that?

          I also see that the patch doesn't include the web ui components. I think postponing them to follow-up work is fine. But if someone wants to test the patch on a real cluster, can you please highlight the steps needed to setup a cluster using the new fair scheduler and if there are changes in setup compared to the old MR1 version.

          Show
          Ahmed Radwan added a comment - Thanks Patrick. The patch successfully applies and compiles on trunk. The included new tests also successfully run. I'll postpone specific code comments, since I have some basic questions: The old fair scheduler used slots as the basic allocation units, and calculating shares and other features (like preemption) relied on this model. In MR2 we have containers with variable resource attributes. Currently the only supported attribute is memory, but it is planned to add more attributes (network bandwidth, cpu, etc.). So it is not clear for me how this mapping is handled. Are you envisioning combing such attributes for calculating shares, how is this done? and how preemption will be done in such case, for example in the old MR1, the most recent slot was killed. But now slots are not equal and this complicate the choices. I am just using preemption as an example. Can you please elaborate on that? I also see that the patch doesn't include the web ui components. I think postponing them to follow-up work is fine. But if someone wants to test the patch on a real cluster, can you please highlight the steps needed to setup a cluster using the new fair scheduler and if there are changes in setup compared to the old MR1 version.
          Hide
          NO NAME added a comment -

          This patch should apply cleanly to trunk, thanks.

          Show
          NO NAME added a comment - This patch should apply cleanly to trunk, thanks.
          Hide
          Patrick Hunt added a comment -

          Patrick unfortunately while the patch applies to the current trunk, it does not compile. Could you update? Thanks.

          Show
          Patrick Hunt added a comment - Patrick unfortunately while the patch applies to the current trunk, it does not compile. Could you update? Thanks.
          Hide
          NO NAME added a comment -

          This is patch aggregating all sub-ticket patches and also bringing up to date with trunk.

          This patch is ready for review, thanks.

          Show
          NO NAME added a comment - This is patch aggregating all sub-ticket patches and also bringing up to date with trunk. This patch is ready for review, thanks.
          Hide
          NO NAME added a comment -

          This JIRA has been chunked up into smaller JIRAs to make the patch submission and review more sane.

          I've attached a patch for the first of these sub-tasks.
          https://issues.apache.org/jira/browse/MAPREDUCE-3600

          Show
          NO NAME added a comment - This JIRA has been chunked up into smaller JIRAs to make the patch submission and review more sane. I've attached a patch for the first of these sub-tasks. https://issues.apache.org/jira/browse/MAPREDUCE-3600
          Hide
          NO NAME added a comment -

          @Mahedev and others

          A follow up question. In the MR1 code, there was a way to assign a priority to a specific job. This let users whose jobs all go to the same queue give more importance to certain jobs over others and it is used in the fair scheduler.

          The MR2 code does seem to have similar support, i.e. per-AM priorities. For instance, in ApplicationSubmissionContext we see:

           
            /**
             * Set the <code>Priority</code> of the application.
             * @param priority <code>Priority</code> of the application
             */
            @Public
            @Stable
            public void setPriority(Priority priority);
          

          The MR AM also seems to support per-job prioritization, though it doesn't look like this code is called anywhere.

            /**
             * Set the priority of a running job.
             * @param priority the new priority for the job.
             * @throws IOException
             */
            public void setPriority(JobPriority priority) 
                throws IOException, InterruptedException {
          

          Such priorities are not, however, passed downstream to the scheduler as far as I can see (they aren't reflected in SchedulerApp class). I can just patch the code to pass these to the scheduler, but I'm curious whether there is something I'm missing here, especially given the last comment which seems to contradict that priorities can be applied across jobs.

          Show
          NO NAME added a comment - @Mahedev and others A follow up question. In the MR1 code, there was a way to assign a priority to a specific job. This let users whose jobs all go to the same queue give more importance to certain jobs over others and it is used in the fair scheduler. The MR2 code does seem to have similar support, i.e. per-AM priorities. For instance, in ApplicationSubmissionContext we see: /** * Set the <code>Priority</code> of the application. * @param priority <code>Priority</code> of the application */ @Public @Stable public void setPriority(Priority priority); The MR AM also seems to support per-job prioritization, though it doesn't look like this code is called anywhere. /** * Set the priority of a running job. * @param priority the new priority for the job. * @throws IOException */ public void setPriority(JobPriority priority) throws IOException, InterruptedException { Such priorities are not , however, passed downstream to the scheduler as far as I can see (they aren't reflected in SchedulerApp class). I can just patch the code to pass these to the scheduler, but I'm curious whether there is something I'm missing here, especially given the last comment which seems to contradict that priorities can be applied across jobs.
          Hide
          Mahadev konar added a comment -

          @Patrick,
          Yes the priorities are for the requests by a single AM. There are no priorities across jobs(/AM's). A DAG of MR jobs in a single AM is very much a possibility and we have been talking to PIG/Oozie folks for such a possibility, but as of now an AM maps to a single job.

          Show
          Mahadev konar added a comment - @Patrick, Yes the priorities are for the requests by a single AM. There are no priorities across jobs(/AM's). A DAG of MR jobs in a single AM is very much a possibility and we have been talking to PIG/Oozie folks for such a possibility, but as of now an AM maps to a single job.
          Hide
          Todd Lipcon added a comment -

          Hi Patrick. Yes, there is an AM-per-job in the current implementation. There's some folks I know looking at bundling full workflows into a single AM, but they would still be fairly short-lived and associated with a single submitting user.

          Show
          Todd Lipcon added a comment - Hi Patrick. Yes, there is an AM-per-job in the current implementation. There's some folks I know looking at bundling full workflows into a single AM, but they would still be fairly short-lived and associated with a single submitting user.
          Hide
          NO NAME added a comment -

          Ah, perhaps my issue is a broader misunderstanding of the MR2 architecture.

          I thought the Application Master was a long lived service for a particular application (given its name). I.e. for MapReduce you would expect to have a single AM running and coordinating submissions for several jobs. Your last sentence seems to suggest this is incorrect. Is a new AM instantiated for each MR job?

          Also, you mentioned priorities. I can't find any details on the scheduling semantic associated with priorities. I'm assuming those are supposed to convey the scheduler ordering for requests from a particular AM? I.e. all priority 1 containers should be scheduled before the first priority 2 container? This made no sense to me before, but it makes more sense if there is a new AM for each job.

          Show
          NO NAME added a comment - Ah, perhaps my issue is a broader misunderstanding of the MR2 architecture. I thought the Application Master was a long lived service for a particular application (given its name). I.e. for MapReduce you would expect to have a single AM running and coordinating submissions for several jobs. Your last sentence seems to suggest this is incorrect. Is a new AM instantiated for each MR job? Also, you mentioned priorities. I can't find any details on the scheduling semantic associated with priorities. I'm assuming those are supposed to convey the scheduler ordering for requests from a particular AM? I.e. all priority 1 containers should be scheduled before the first priority 2 container? This made no sense to me before, but it makes more sense if there is a new AM for each job.
          Hide
          NO NAME added a comment -

          Ah, perhaps my issue is a broader misunderstanding of the MR2 architecture.

          I thought the Application Master was a long lived service for a particular application (given its name). I.e. for MapReduce you would expect to have a single AM running and coordinating submissions for several jobs. Your last sentence seems to suggest this is incorrect. Is a new AM instantiated for each MR job?

          Also, you mentioned priorities. I can't find any details on the scheduling semantic associated with priorities. I'm assuming those are supposed to convey the scheduler ordering for requests from a particular AM? I.e. all priority 1 containers should be scheduled before the first priority 2 container? This made no sense to me before, but it makes more sense if there is a new AM for each job.

          Show
          NO NAME added a comment - Ah, perhaps my issue is a broader misunderstanding of the MR2 architecture. I thought the Application Master was a long lived service for a particular application (given its name). I.e. for MapReduce you would expect to have a single AM running and coordinating submissions for several jobs. Your last sentence seems to suggest this is incorrect. Is a new AM instantiated for each MR job? Also, you mentioned priorities. I can't find any details on the scheduling semantic associated with priorities. I'm assuming those are supposed to convey the scheduler ordering for requests from a particular AM? I.e. all priority 1 containers should be scheduled before the first priority 2 container? This made no sense to me before, but it makes more sense if there is a new AM for each job.
          Hide
          Mahadev konar added a comment -

          Patrick,
          I'd say it could do both. If you look at the Capacity Scheduler, it does inter application scheduling (deciding on which application gets a resource) and then also decides which priority request within the application should be satisfied first (in CS its the highest first).

          Take a look at

          LeafQueue.assignContainers()
          ParentQueue.assignContainers()
          

          As for the user information, its is available in the SchedulerApp.getUser().

          Also, a single AM amounts to a single job currently, I am not sure what you mean by "jobs requested for a single AM".

          Hope that helps.

          Show
          Mahadev konar added a comment - Patrick, I'd say it could do both. If you look at the Capacity Scheduler, it does inter application scheduling (deciding on which application gets a resource) and then also decides which priority request within the application should be satisfied first (in CS its the highest first). Take a look at LeafQueue.assignContainers() ParentQueue.assignContainers() As for the user information, its is available in the SchedulerApp.getUser(). Also, a single AM amounts to a single job currently, I am not sure what you mean by "jobs requested for a single AM". Hope that helps.
          Hide
          NO NAME added a comment -

          I have a few questions about the way that schedulers are implemented in MR2.

          It's not clear to me whether the Scheduler class used by the Resource Manager is responsible for only inter-application scheduling or both inter- ant intra-application scheduling. It looks to me like only the former is supported, because the resource requests (via allocate()) do not give any meta-data regarding the user initiating the request. So if we wanted to enforce Fair Sharing for jobs requested from a single AM, this doesn't seem possible. When I look at the Capacity Scheduler, for instance, it seems like allocation requests are assigned to queues based only on the application from which they are made, not the actual user launching the job.

          Is the idea that, in a shared cluster, each administrative unit has its own MR Application Master running?

          I have some follow-up but first want to see if this is actually a correct interpretation of the design spec.

          Show
          NO NAME added a comment - I have a few questions about the way that schedulers are implemented in MR2. It's not clear to me whether the Scheduler class used by the Resource Manager is responsible for only inter-application scheduling or both inter- ant intra-application scheduling. It looks to me like only the former is supported, because the resource requests (via allocate()) do not give any meta-data regarding the user initiating the request. So if we wanted to enforce Fair Sharing for jobs requested from a single AM, this doesn't seem possible. When I look at the Capacity Scheduler, for instance, it seems like allocation requests are assigned to queues based only on the application from which they are made, not the actual user launching the job. Is the idea that, in a shared cluster, each administrative unit has its own MR Application Master running? I have some follow-up but first want to see if this is actually a correct interpretation of the design spec.
          Hide
          Arun C Murthy added a comment -

          Thanks for taking this up Patrick. Let me know if you need help.

          Show
          Arun C Murthy added a comment - Thanks for taking this up Patrick. Let me know if you need help.

            People

            • Assignee:
              NO NAME
              Reporter:
              NO NAME
            • Votes:
              0 Vote for this issue
              Watchers:
              39 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development