Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-3453

Fair Scheduler: Parts of preemption logic uses DefaultResourceCalculator even in DRF mode causing thrashing

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.6.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: fairscheduler
    • Labels:
      None

      Description

      There are two places in preemption code flow where DefaultResourceCalculator is used, even in DRF mode.
      Which basically results in more resources getting preempted than needed, and those extra preempted containers aren’t even getting to the “starved” queue since scheduling logic is based on DRF's Calculator.

      Following are the two places :
      1.

      FSLeafQueue.java
      private boolean isStarved(Resource share)
      

      A queue shouldn’t be marked as “starved” if the dominant resource usage
      is >= fair/minshare.

      2.

      FairScheduler.java
      protected Resource resToPreempt(FSLeafQueue sched, long curTime)
      

      --------------------------------------------------------------

      One more thing that I believe needs to change in DRF mode is : during a preemption round,if preempting a few containers results in satisfying needs of a resource type, then we should exit that preemption round, since the containers that we just preempted should bring the dominant resource usage to min/fair share.

      1. YARN-3453.1.patch
        6 kB
        Arun Suresh
      2. YARN-3453.2.patch
        14 kB
        Arun Suresh
      3. YARN-3453.3.patch
        22 kB
        Arun Suresh
      4. YARN-3453.4.patch
        30 kB
        Arun Suresh
      5. YARN-3453.5.patch
        32 kB
        Arun Suresh

        Issue Links

          Activity

          Hide
          peng.zhang Peng Zhang added a comment -

          Ashwin Shankar
          I‘ve the same problem in our cluster. I think 2 points should be done to make it work:

          1. Calculator should be configurable, so isStarved() can use DominantResourceCalculator to test. But this still has corner case: cluster(40960, 8), expected( 16384, 4), real(4096, 4) it still result with "starved". My idea for this is to lower starve threshold to ignore this. Any good suggestions?
          2. in resToPreempt(), we should change logic of calculation, changing calculator is not enough. My hacked code is like below:
            -      Resource target = Resources.min(getResourceCalculator(), clusterResource,
            -          sched.getFairShare(), sched.getDemand());
            +      Resource resourceUpperBound = Resources.componentwiseMin(
                       sched.getFairShare(), sched.getDemand()); // min value of both cpu and memory as upperBound of request resource
            +      float targetRatio = 0;
            +        // getResourceAsValue is not super method in ResourceCalculator, and i cannot figure out one name for this calculation logic, so hack like this
            +      if (resourceCalculator instanceof DominantResourceCalculator) {
            +        // the ratio of demand can be get under resourceUpperBound.
            +        targetRatio = ((DominantResourceCalculator)resourceCalculator)
            +            .getResourceAsValue(sched.getDemand(), resourceUpperBound, false);  // min of ratio for cpu and memory
            +      } else {
            +        targetRatio = Resources.ratio(resourceCalculator, sched.getDemand(),
            +            resourceUpperBound);
            +      }
            +      Resource target = Resources.multiply(sched.getDemand(), targetRatio);  // demand resource can be fulfilled under fair share. 
            

          Besides "drf" policy problem with preemption, I also filed YARN-3405 to describe some common problems with preemption.

          Show
          peng.zhang Peng Zhang added a comment - Ashwin Shankar I‘ve the same problem in our cluster. I think 2 points should be done to make it work: Calculator should be configurable, so isStarved() can use DominantResourceCalculator to test. But this still has corner case: cluster(40960, 8), expected( 16384, 4), real(4096, 4) it still result with "starved". My idea for this is to lower starve threshold to ignore this. Any good suggestions? in resToPreempt(), we should change logic of calculation, changing calculator is not enough. My hacked code is like below: - Resource target = Resources.min(getResourceCalculator(), clusterResource, - sched.getFairShare(), sched.getDemand()); + Resource resourceUpperBound = Resources.componentwiseMin( sched.getFairShare(), sched.getDemand()); // min value of both cpu and memory as upperBound of request resource + float targetRatio = 0; + // getResourceAsValue is not super method in ResourceCalculator, and i cannot figure out one name for this calculation logic, so hack like this + if (resourceCalculator instanceof DominantResourceCalculator) { + // the ratio of demand can be get under resourceUpperBound. + targetRatio = ((DominantResourceCalculator)resourceCalculator) + .getResourceAsValue(sched.getDemand(), resourceUpperBound, false ); // min of ratio for cpu and memory + } else { + targetRatio = Resources.ratio(resourceCalculator, sched.getDemand(), + resourceUpperBound); + } + Resource target = Resources.multiply(sched.getDemand(), targetRatio); // demand resource can be fulfilled under fair share. Besides "drf" policy problem with preemption, I also filed YARN-3405 to describe some common problems with preemption.
          Hide
          ashwinshankar77 Ashwin Shankar added a comment -

          Peng Zhang,

          Calculator should be configurable, so isStarved() can use DominantResourceCalculator to test. But this still has corner case: cluster(40960, 8), expected( 16384, 4), real(4096, 4) it still result with "starved". My idea for this is to lower starve threshold to ignore this. Any good suggestions?

          Would it help if we have a separate method or modify DominantResourceCalculator.compare() (in a backward compatible way), such that if the dominant
          resource is equal, then return -1. Basically NOT do the following while checking for starvation :

          else {
                l = getResourceAsValue(clusterResource, lhs, false);
                r = getResourceAsValue(clusterResource, rhs, false);
                if (l < r) {
                  return -1;
                } else if (l > r) {
                  return 1;
                }
              }
          

          in resToPreempt(), we should change logic of calculation, changing calculator is not enough.

          Doing componentwiseMin makes sense. But why should we calculate targetRatio ? Shouldnt we just preempt (resourceUpperBound - usage) ?
          Can you pls give an example ?

          Show
          ashwinshankar77 Ashwin Shankar added a comment - Peng Zhang , Calculator should be configurable, so isStarved() can use DominantResourceCalculator to test. But this still has corner case: cluster(40960, 8), expected( 16384, 4), real(4096, 4) it still result with "starved". My idea for this is to lower starve threshold to ignore this. Any good suggestions? Would it help if we have a separate method or modify DominantResourceCalculator.compare() (in a backward compatible way), such that if the dominant resource is equal, then return -1. Basically NOT do the following while checking for starvation : else { l = getResourceAsValue(clusterResource, lhs, false ); r = getResourceAsValue(clusterResource, rhs, false ); if (l < r) { return -1; } else if (l > r) { return 1; } } in resToPreempt(), we should change logic of calculation, changing calculator is not enough. Doing componentwiseMin makes sense. But why should we calculate targetRatio ? Shouldnt we just preempt (resourceUpperBound - usage) ? Can you pls give an example ?
          Hide
          peng.zhang Peng Zhang added a comment -

          Ashwin Shankar

          Would it help if we have a separate method or modify DominantResourceCalculator.compare() (in a backward compatible way), such that if the dominant resource is equal, then return -1.

          Changing compare() to fix this corner case may cause some conflict with "dominant" notion.
          I found some operations in DominantResourceCalculator now are already not be consistent with "Dominant" notion. I think they are more related with multiple resources calculator.
          So I think maybe we'd better extract new layer for these operations. One layer for resource dimension(only memory or multiple resources), and the other for policy( fair or dominant, and maybe other policy).

          Doing componentwiseMin makes sense. But why should we calculate targetRatio ? Shouldnt we just preempt (resourceUpperBound - usage) ? Can you pls give an example ?

          eg: fairshare (100g, 3core) demand (10g, 10core), componentwiseMin return (10g, 3core).
          When do preemption, assume each preempted container is (1g, 1core).
          After preemping 3 containers, toPreempt is (7g, 0core), still bigger than None(0g, 0core) even using Dominant calculator. So it will still preempt other 7 containers(7g, 7core) which cannot be scheduled. This cause waste of preemption.
          To decrease this kind of waste, I want to found what's the ratio of demand can be fulfilled by resourceUpperBound, and use this ratio * resourceUpperBound to be targetResource.

          Actually, current implementation ignored the resource boundary of each requested container, so even after above logic, it still will has some waste.

          Show
          peng.zhang Peng Zhang added a comment - Ashwin Shankar Would it help if we have a separate method or modify DominantResourceCalculator.compare() (in a backward compatible way), such that if the dominant resource is equal, then return -1. Changing compare() to fix this corner case may cause some conflict with "dominant" notion. I found some operations in DominantResourceCalculator now are already not be consistent with "Dominant" notion. I think they are more related with multiple resources calculator. So I think maybe we'd better extract new layer for these operations. One layer for resource dimension(only memory or multiple resources), and the other for policy( fair or dominant, and maybe other policy). Doing componentwiseMin makes sense. But why should we calculate targetRatio ? Shouldnt we just preempt (resourceUpperBound - usage) ? Can you pls give an example ? eg: fairshare (100g, 3core) demand (10g, 10core), componentwiseMin return (10g, 3core). When do preemption, assume each preempted container is (1g, 1core). After preemping 3 containers, toPreempt is (7g, 0core), still bigger than None(0g, 0core) even using Dominant calculator. So it will still preempt other 7 containers(7g, 7core) which cannot be scheduled. This cause waste of preemption. To decrease this kind of waste, I want to found what's the ratio of demand can be fulfilled by resourceUpperBound, and use this ratio * resourceUpperBound to be targetResource. Actually, current implementation ignored the resource boundary of each requested container, so even after above logic, it still will has some waste.
          Hide
          peng.zhang Peng Zhang added a comment -

          Update code snippet for calculation of resDueToFairShare

                Resource target;
                if (resourceCalculator instanceof DominantResourceCalculator) {
                  float targetRatio = Math.min(1,
                      ((DominantResourceCalculator) resourceCalculator)
                      .getResourceAsValue(sched.getDemand(), resourceUpperBound, false));
                  target = Resources.multiply(sched.getDemand(), targetRatio);
                } else {
                  target = resourceUpperBound;
                }
                resDueToFairShare = Resources.max(resourceCalculator, clusterResource,
                    Resources.none(), Resources.subtract(target, sched.getResourceUsage()));
          
          Show
          peng.zhang Peng Zhang added a comment - Update code snippet for calculation of resDueToFairShare Resource target; if (resourceCalculator instanceof DominantResourceCalculator) { float targetRatio = Math .min(1, ((DominantResourceCalculator) resourceCalculator) .getResourceAsValue(sched.getDemand(), resourceUpperBound, false )); target = Resources.multiply(sched.getDemand(), targetRatio); } else { target = resourceUpperBound; } resDueToFairShare = Resources.max(resourceCalculator, clusterResource, Resources.none(), Resources.subtract(target, sched.getResourceUsage()));
          Hide
          asuresh Arun Suresh added a comment -

          Peng Zhang, Ashwin Shankar,
          Thank you reporting this.. and the associated discussion

          I vote that we :

          1. fix the isStarved() method to use the correct Calculator
          2. fix the resToPreempt() method to use componentWiseMin for the target... but defer using the targetRatio, since it is probably an optimization and can be addressed in a future JIRA

          I have attached a preliminary patch that does this..
          Will upload one with test cases shortly

          Show
          asuresh Arun Suresh added a comment - Peng Zhang , Ashwin Shankar , Thank you reporting this.. and the associated discussion I vote that we : fix the isStarved() method to use the correct Calculator fix the resToPreempt() method to use componentWiseMin for the target... but defer using the targetRatio , since it is probably an optimization and can be addressed in a future JIRA I have attached a preliminary patch that does this.. Will upload one with test cases shortly
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 15m 55s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 javac 7m 37s There were no new javac warning messages.
          +1 javadoc 9m 35s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 46s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 33s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 1m 25s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 50m 14s Tests passed in hadoop-yarn-server-resourcemanager.
              88m 10s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12737319/YARN-3453.1.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / c59e745
          hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8179/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8179/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8179/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 15m 55s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 37s There were no new javac warning messages. +1 javadoc 9m 35s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 46s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 1m 25s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 50m 14s Tests passed in hadoop-yarn-server-resourcemanager.     88m 10s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12737319/YARN-3453.1.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / c59e745 hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8179/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8179/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8179/console This message was automatically generated.
          Hide
          kasha Karthik Kambatla added a comment -

          Should we add SchedulingPolicy#getResourceCalculator() and use that instead?

          Show
          kasha Karthik Kambatla added a comment - Should we add SchedulingPolicy#getResourceCalculator() and use that instead?
          Hide
          asuresh Arun Suresh added a comment -

          Agreed..
          Updating patch with your suggestion..

          Show
          asuresh Arun Suresh added a comment - Agreed.. Updating patch with your suggestion..
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 15m 7s Findbugs (version ) appears to be broken on trunk.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 javac 7m 32s There were no new javac warning messages.
          +1 javadoc 9m 39s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 30s There were no new checkstyle issues.
          +1 whitespace 0m 1s The patch has no lines that end in whitespace.
          +1 install 1m 35s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 1m 25s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 50m 7s Tests passed in hadoop-yarn-server-resourcemanager.
              86m 58s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12737386/YARN-3453.2.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / bc85959
          hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8187/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8187/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8187/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 15m 7s Findbugs (version ) appears to be broken on trunk. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 32s There were no new javac warning messages. +1 javadoc 9m 39s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 30s There were no new checkstyle issues. +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 35s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 1m 25s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 50m 7s Tests passed in hadoop-yarn-server-resourcemanager.     86m 58s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12737386/YARN-3453.2.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / bc85959 hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8187/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8187/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8187/console This message was automatically generated.
          Hide
          kasha Karthik Kambatla added a comment -

          Few comments:

          1. New imports in FairScheduler and FSLeafQueue are not required.
          2. Looking at the remaining uses of DefaultResourceCalculator in FairScheduler, we could benefit from updating all of them to DominantResourceCalculator? Ashwin Shankar - do you concur?
          3. In FairScheduler, changing the scope of RESOURCE_CALCULATOR and DOMINANT_RESOURCE_CALCULATOR is not required.
          4. We should add unit-tests to avoid regressions in the future.
          5. Nit: In each of the policies, my preference would be not make the calculator and comparator members static unless required. We have had cases where our tests would invoke multiple instances of the class leading to issues. Not that I foresee multiple instantiations for these classes, but would like to avoid it if we can.
          Show
          kasha Karthik Kambatla added a comment - Few comments: New imports in FairScheduler and FSLeafQueue are not required. Looking at the remaining uses of DefaultResourceCalculator in FairScheduler, we could benefit from updating all of them to DominantResourceCalculator? Ashwin Shankar - do you concur? In FairScheduler, changing the scope of RESOURCE_CALCULATOR and DOMINANT_RESOURCE_CALCULATOR is not required. We should add unit-tests to avoid regressions in the future. Nit: In each of the policies, my preference would be not make the calculator and comparator members static unless required. We have had cases where our tests would invoke multiple instances of the class leading to issues. Not that I foresee multiple instantiations for these classes, but would like to avoid it if we can.
          Hide
          ashwinshankar77 Ashwin Shankar added a comment -

          hey folks, Looking into the patch, will get back with comments.

          Show
          ashwinshankar77 Ashwin Shankar added a comment - hey folks, Looking into the patch, will get back with comments.
          Hide
          ashwinshankar77 Ashwin Shankar added a comment -

          hey Arun,
          Thanks for working on this !
          Couple more comments in addition to Karthik's comments :
          1. Why are we not using componentwisemin here ?
          Resource target = Resources.min(calc, clusterResource,
          sched.getMinShare(), sched.getDemand());

          2. FairScheduler.preemptResources() uses DefaultResourceCalculator and hence would look at only memory.
          This could lead to a problem in the following scenario :
          Preemption round0 : toPreempt = (100G,10 cores)
          ...<we preempt 10 containers of (4G,1 core)
          Preemption round10 : toPreempt = (60G,0 cores)

          In round10, we've satisfied all the cores, the current implementation since its based on DefaultResourceCalculator would continue to preempt
          to satisfy the remaining 60G as well even in DRF, which means we just preempted more cores than we had to.
          Making this calculator DRF wouldnt solve the problem as well, since we would then be preempting less than what is necessary.
          The root of this problem is YARN-2154. I'll leave it upto you to decide what you want to do about this in this jira.

          FairScheduler.java
          while (Resources.greaterThan(RESOURCE_CALCULATOR, clusterResource,
                    toPreempt, Resources.none()))
          

          3. Unit tests needs to be added.

          @Karthik Kambatla,

          Looking at the remaining uses of DefaultResourceCalculator in FairScheduler, we could benefit from updating all of them to DominantResourceCalculator? Ashwin Shankar - do you concur?

          Overall I see it would be beneficial. But I'm not so sure if the callers of FairScheduler.getResourceCalculator would be okay with getting DominantResourceCalculator always? I see its mostly called my Fair Reservation System feature.

          Show
          ashwinshankar77 Ashwin Shankar added a comment - hey Arun, Thanks for working on this ! Couple more comments in addition to Karthik's comments : 1. Why are we not using componentwisemin here ? Resource target = Resources.min(calc, clusterResource, sched.getMinShare(), sched.getDemand()); 2. FairScheduler.preemptResources() uses DefaultResourceCalculator and hence would look at only memory. This could lead to a problem in the following scenario : Preemption round0 : toPreempt = (100G,10 cores) ...<we preempt 10 containers of (4G,1 core) Preemption round10 : toPreempt = (60G,0 cores) In round10, we've satisfied all the cores, the current implementation since its based on DefaultResourceCalculator would continue to preempt to satisfy the remaining 60G as well even in DRF, which means we just preempted more cores than we had to. Making this calculator DRF wouldnt solve the problem as well, since we would then be preempting less than what is necessary. The root of this problem is YARN-2154 . I'll leave it upto you to decide what you want to do about this in this jira. FairScheduler.java while (Resources.greaterThan(RESOURCE_CALCULATOR, clusterResource, toPreempt, Resources.none())) 3. Unit tests needs to be added. @ Karthik Kambatla , Looking at the remaining uses of DefaultResourceCalculator in FairScheduler, we could benefit from updating all of them to DominantResourceCalculator? Ashwin Shankar - do you concur? Overall I see it would be beneficial. But I'm not so sure if the callers of FairScheduler.getResourceCalculator would be okay with getting DominantResourceCalculator always? I see its mostly called my Fair Reservation System feature.
          Hide
          asuresh Arun Suresh added a comment -

          Apologize for the delay.
          Thanks for the review Ashwin Shankar.

          Why are we not using componentwisemin here ?

          Yup.. this can be changed to componentwisemin.

          FairScheduler.preemptResources() uses DefaultResourceCalculator and hence would look at only memory.

          Agree.. And as you mentioned, think we will have to fix YARN-2154 too.

          Show
          asuresh Arun Suresh added a comment - Apologize for the delay. Thanks for the review Ashwin Shankar . Why are we not using componentwisemin here ? Yup.. this can be changed to componentwisemin. FairScheduler.preemptResources() uses DefaultResourceCalculator and hence would look at only memory. Agree.. And as you mentioned, think we will have to fix YARN-2154 too.
          Hide
          asuresh Arun Suresh added a comment -

          Uploading updated patch :

          • Added unit-tests
          • clean-up code based on comments

          Karthik Kambatla,

          Nit: In each of the policies, my preference would be not make the calculator and comparator members static unless required. We have had cases where our tests would invoke multiple instances of the class leading to issues. Not that I foresee multiple instantiations for these classes, but would like to avoid it if we can.

          If it ok with you, I feel we should infact make it static. Am of the opinion that the code reads better, is a lot cleaner and efficient, since only 1 instance is ever created.. We are always at the liberty to over-ride the getComparator/Calculator method in test (and possible subclasses)

          .. think we will have to fix YARN-2154 too.

          On further thought.. and after consultation with Karthik Kambatla, Think we can decouple from that JIRA, given its larger scope.

          Show
          asuresh Arun Suresh added a comment - Uploading updated patch : Added unit-tests clean-up code based on comments Karthik Kambatla , Nit: In each of the policies, my preference would be not make the calculator and comparator members static unless required. We have had cases where our tests would invoke multiple instances of the class leading to issues. Not that I foresee multiple instantiations for these classes, but would like to avoid it if we can. If it ok with you, I feel we should infact make it static. Am of the opinion that the code reads better, is a lot cleaner and efficient, since only 1 instance is ever created.. We are always at the liberty to over-ride the getComparator/Calculator method in test (and possible subclasses) .. think we will have to fix YARN-2154 too. On further thought.. and after consultation with Karthik Kambatla , Think we can decouple from that JIRA, given its larger scope.
          Hide
          asuresh Arun Suresh added a comment -

          New Patch :

          • Cleaned up some doc
          • changed the name of resToPreempt to resourceDeficit. I feel resToPreempt is not just confusing but kinda wrong.. given that the method technically does not find resources to Preempt from the given queue. It actually finds the resource deficit that would bring it back to min/fair share.
          Show
          asuresh Arun Suresh added a comment - New Patch : Cleaned up some doc changed the name of resToPreempt to resourceDeficit . I feel resToPreempt is not just confusing but kinda wrong.. given that the method technically does not find resources to Preempt from the given queue. It actually finds the resource deficit that would bring it back to min/fair share.
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 16m 5s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
          +1 javac 7m 43s There were no new javac warning messages.
          +1 javadoc 9m 34s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 47s There were no new checkstyle issues.
          +1 whitespace 0m 6s The patch has no lines that end in whitespace.
          +1 install 1m 20s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 1m 27s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 50m 58s Tests passed in hadoop-yarn-server-resourcemanager.
              89m 1s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12744446/YARN-3453.4.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 63d0365
          hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8474/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8474/testReport/
          Java 1.7.0_55
          uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8474/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 5s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 7m 43s There were no new javac warning messages. +1 javadoc 9m 34s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 47s There were no new checkstyle issues. +1 whitespace 0m 6s The patch has no lines that end in whitespace. +1 install 1m 20s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 1m 27s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 50m 58s Tests passed in hadoop-yarn-server-resourcemanager.     89m 1s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12744446/YARN-3453.4.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 63d0365 hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8474/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8474/testReport/ Java 1.7.0_55 uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8474/console This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 16m 0s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
          +1 javac 7m 41s There were no new javac warning messages.
          +1 javadoc 9m 37s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 44s There were no new checkstyle issues.
          +1 whitespace 0m 6s The patch has no lines that end in whitespace.
          +1 install 1m 23s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 1m 26s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          -1 yarn tests 61m 14s Tests failed in hadoop-yarn-server-resourcemanager.
              99m 10s  



          Reason Tests
          Timed out tests org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12744446/YARN-3453.4.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 63d0365
          hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8473/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8473/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8473/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 0s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 7m 41s There were no new javac warning messages. +1 javadoc 9m 37s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 44s There were no new checkstyle issues. +1 whitespace 0m 6s The patch has no lines that end in whitespace. +1 install 1m 23s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 1m 26s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 yarn tests 61m 14s Tests failed in hadoop-yarn-server-resourcemanager.     99m 10s   Reason Tests Timed out tests org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12744446/YARN-3453.4.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 63d0365 hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8473/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8473/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8473/console This message was automatically generated.
          Hide
          asuresh Arun Suresh added a comment -

          Test case failure is un-related. Jenkins had passed when i kicked it off manually here

          Show
          asuresh Arun Suresh added a comment - Test case failure is un-related. Jenkins had passed when i kicked it off manually here
          Hide
          peng.zhang Peng Zhang added a comment -

          Thanks Arun Suresh for working on this

          comments:

          1. Why not changing all usage of calculator in FairScheduler to policy related. In below code, RESOURCE_CALCULATOR only calculate memory, and it may return false when resToPreempt is (0, non-zero) for DRF policy:
            FairScheduler.java
                 if (Resources.greaterThan(RESOURCE_CALCULATOR, clusterResource, resToPreempt,
                     Resources.none())) {
            
          Show
          peng.zhang Peng Zhang added a comment - Thanks Arun Suresh for working on this comments: Why not changing all usage of calculator in FairScheduler to policy related. In below code, RESOURCE_CALCULATOR only calculate memory, and it may return false when resToPreempt is (0, non-zero) for DRF policy: FairScheduler.java if (Resources.greaterThan(RESOURCE_CALCULATOR, clusterResource, resToPreempt, Resources.none())) {
          Hide
          asuresh Arun Suresh added a comment -

          Peng Zhang, thanks for the review..

          Why not changing all usage of calculator in FairScheduler to policy related..

          The main reason why I left that section of the code to use RESOURCE_CALCULATOR is that at that point, we do not have the associated queue information.. and thus we will not know the relevant policy required.. and hence will not be able to pick the correct calculator..

          But after you pointed it out.. I gave it some further thought.. and have uploaded a new patch : YARN-3453.5.patch .. where I do not use a calculator.. but instead I check if either mem or vcores > 0.

          Do take a look and let me know if its fine..

          But in any case, as Ashwin Shankar had pointed out, we need a proper re-look at YARN-2154 to improve pre-emption in general.

          Show
          asuresh Arun Suresh added a comment - Peng Zhang , thanks for the review.. Why not changing all usage of calculator in FairScheduler to policy related.. The main reason why I left that section of the code to use RESOURCE_CALCULATOR is that at that point, we do not have the associated queue information.. and thus we will not know the relevant policy required.. and hence will not be able to pick the correct calculator.. But after you pointed it out.. I gave it some further thought.. and have uploaded a new patch : YARN-3453 .5.patch .. where I do not use a calculator.. but instead I check if either mem or vcores > 0. Do take a look and let me know if its fine.. But in any case, as Ashwin Shankar had pointed out, we need a proper re-look at YARN-2154 to improve pre-emption in general.
          Hide
          peng.zhang Peng Zhang added a comment -

          I understood your thought. My suggestion is based on our practice: I found it's confusing to use different policy in queue configuration: eg. parent use fair, child use drf may cause child queue has no resource on cpu dimension, so job will hang there. So we use only drf in one cluster, and change the code to support setting the calculator class in scheduler scope.

          After review above comments, I am reminded that the case (0 GB, non-zero cores) like (non-zero GB, 0 cores) will also cause preempt more resources than necessary.

          I mentioned before:

          To decrease this kind of waste, I want to found what's the ratio of demand can be fulfilled by resourceUpperBound, and use this ratio * resourceUpperBound to be targetResource.

          Actually, current implementation ignored the resource boundary of each requested container, so even after above logic, it still will has some waste.

          As for YARN-2154, if we want to only preempt containers can satisfy incoming request, IMHO, we should to do preemption for each incoming request instead count them up with resourceDeficit.

          Show
          peng.zhang Peng Zhang added a comment - I understood your thought. My suggestion is based on our practice: I found it's confusing to use different policy in queue configuration: eg. parent use fair, child use drf may cause child queue has no resource on cpu dimension, so job will hang there. So we use only drf in one cluster, and change the code to support setting the calculator class in scheduler scope. After review above comments, I am reminded that the case (0 GB, non-zero cores) like (non-zero GB, 0 cores) will also cause preempt more resources than necessary. I mentioned before: To decrease this kind of waste, I want to found what's the ratio of demand can be fulfilled by resourceUpperBound, and use this ratio * resourceUpperBound to be targetResource. Actually, current implementation ignored the resource boundary of each requested container, so even after above logic, it still will has some waste. As for YARN-2154 , if we want to only preempt containers can satisfy incoming request, IMHO, we should to do preemption for each incoming request instead count them up with resourceDeficit .
          Hide
          asuresh Arun Suresh added a comment -

          Peng Zhang,

          After review above comments, I am reminded that the case (0 GB, non-zero cores) like (non-zero GB, 0 cores) will also cause preempt more resources than necessary.

          I agree... But I feel instead of fixing it here, if we can have a comprehensive fix as requested by YARN-2154 ( Karthik Kambatla and myself had an offline discussion about how we should actually break from the preemption loop when incoming requests are satisfied), then we wont even hit this case.
          Further more, this JIRA fixes the isStarved() method in the Queue correctly, so at the very least, the toPreempt resource object would be smaller (and thus would implicitly result in less pre-emptions)

          I also agree fining the ratio of demand is definitely useful. But again, let us grab all the low hanging fruit first. I propose we create a separate JIRA for that.

          Show
          asuresh Arun Suresh added a comment - Peng Zhang , After review above comments, I am reminded that the case (0 GB, non-zero cores) like (non-zero GB, 0 cores) will also cause preempt more resources than necessary. I agree... But I feel instead of fixing it here, if we can have a comprehensive fix as requested by YARN-2154 ( Karthik Kambatla and myself had an offline discussion about how we should actually break from the preemption loop when incoming requests are satisfied), then we wont even hit this case. Further more, this JIRA fixes the isStarved() method in the Queue correctly, so at the very least, the toPreempt resource object would be smaller (and thus would implicitly result in less pre-emptions) I also agree fining the ratio of demand is definitely useful. But again, let us grab all the low hanging fruit first. I propose we create a separate JIRA for that.
          Hide
          peng.zhang Peng Zhang added a comment -

          Yes, I agree with more fix in separate JIRA. And YARN-3453.5.patch LGTM.

          Show
          peng.zhang Peng Zhang added a comment - Yes, I agree with more fix in separate JIRA. And YARN-3453 .5.patch LGTM.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 16m 4s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
          +1 javac 7m 47s There were no new javac warning messages.
          +1 javadoc 9m 36s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 46s There were no new checkstyle issues.
          +1 whitespace 0m 6s The patch has no lines that end in whitespace.
          +1 install 1m 19s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 1m 25s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          -1 yarn tests 50m 50s Tests failed in hadoop-yarn-server-resourcemanager.
              88m 53s  



          Reason Tests
          Failed unit tests hadoop.yarn.server.resourcemanager.rmapp.TestRMAppTransitions
            hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRMRPCNodeUpdates
            hadoop.yarn.server.resourcemanager.TestResourceTrackerService



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12744652/YARN-3453.5.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / d66302e
          hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8493/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8493/testReport/
          Java 1.7.0_55
          uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8493/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 4s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 7m 47s There were no new javac warning messages. +1 javadoc 9m 36s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 46s There were no new checkstyle issues. +1 whitespace 0m 6s The patch has no lines that end in whitespace. +1 install 1m 19s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 1m 25s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 yarn tests 50m 50s Tests failed in hadoop-yarn-server-resourcemanager.     88m 53s   Reason Tests Failed unit tests hadoop.yarn.server.resourcemanager.rmapp.TestRMAppTransitions   hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRMRPCNodeUpdates   hadoop.yarn.server.resourcemanager.TestResourceTrackerService Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12744652/YARN-3453.5.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / d66302e hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8493/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8493/testReport/ Java 1.7.0_55 uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8493/console This message was automatically generated.
          Hide
          asuresh Arun Suresh added a comment -

          The test case failures are unrelated and they run fine on my laptop.
          Karthik Kambatla, can you please give it a quick rev (only major thing changed apart from addressing your previous comments is addition of test cases)

          Show
          asuresh Arun Suresh added a comment - The test case failures are unrelated and they run fine on my laptop. Karthik Kambatla , can you please give it a quick rev (only major thing changed apart from addressing your previous comments is addition of test cases)
          Hide
          kasha Karthik Kambatla added a comment -

          +1

          Show
          kasha Karthik Kambatla added a comment - +1
          Hide
          asuresh Arun Suresh added a comment -

          Thanks for the reviews Karthik Kambatla, Ashwin Shankar and Peng Zhang
          Will be committing this shortly..

          Show
          asuresh Arun Suresh added a comment - Thanks for the reviews Karthik Kambatla , Ashwin Shankar and Peng Zhang Will be committing this shortly..
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8160 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8160/)
          YARN-3453. Ensure preemption logic in FairScheduler uses DominantResourceCalculator in DRF queues to prevent unnecessary thrashing. (asuresh) (Arun Suresh: rev ac94ba3e185115b83351e35c610c2b8ff91b1ebc)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingPolicy.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSLeafQueue.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FifoPolicy.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/DominantResourceFairnessPolicy.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FairSharePolicy.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8160 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8160/ ) YARN-3453 . Ensure preemption logic in FairScheduler uses DominantResourceCalculator in DRF queues to prevent unnecessary thrashing. (asuresh) (Arun Suresh: rev ac94ba3e185115b83351e35c610c2b8ff91b1ebc) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingPolicy.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSLeafQueue.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FifoPolicy.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/DominantResourceFairnessPolicy.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FairSharePolicy.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #256 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/256/)
          YARN-3453. Ensure preemption logic in FairScheduler uses DominantResourceCalculator in DRF queues to prevent unnecessary thrashing. (asuresh) (Arun Suresh: rev ac94ba3e185115b83351e35c610c2b8ff91b1ebc)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSLeafQueue.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/DominantResourceFairnessPolicy.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FifoPolicy.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FairSharePolicy.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingPolicy.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #256 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/256/ ) YARN-3453 . Ensure preemption logic in FairScheduler uses DominantResourceCalculator in DRF queues to prevent unnecessary thrashing. (asuresh) (Arun Suresh: rev ac94ba3e185115b83351e35c610c2b8ff91b1ebc) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSLeafQueue.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/DominantResourceFairnessPolicy.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FifoPolicy.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FairSharePolicy.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingPolicy.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #986 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/986/)
          YARN-3453. Ensure preemption logic in FairScheduler uses DominantResourceCalculator in DRF queues to prevent unnecessary thrashing. (asuresh) (Arun Suresh: rev ac94ba3e185115b83351e35c610c2b8ff91b1ebc)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSLeafQueue.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FairSharePolicy.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/DominantResourceFairnessPolicy.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingPolicy.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FifoPolicy.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #986 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/986/ ) YARN-3453 . Ensure preemption logic in FairScheduler uses DominantResourceCalculator in DRF queues to prevent unnecessary thrashing. (asuresh) (Arun Suresh: rev ac94ba3e185115b83351e35c610c2b8ff91b1ebc) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSLeafQueue.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FairSharePolicy.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/DominantResourceFairnessPolicy.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingPolicy.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FifoPolicy.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2183 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2183/)
          YARN-3453. Ensure preemption logic in FairScheduler uses DominantResourceCalculator in DRF queues to prevent unnecessary thrashing. (asuresh) (Arun Suresh: rev ac94ba3e185115b83351e35c610c2b8ff91b1ebc)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingPolicy.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FifoPolicy.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSLeafQueue.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/DominantResourceFairnessPolicy.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FairSharePolicy.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2183 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2183/ ) YARN-3453 . Ensure preemption logic in FairScheduler uses DominantResourceCalculator in DRF queues to prevent unnecessary thrashing. (asuresh) (Arun Suresh: rev ac94ba3e185115b83351e35c610c2b8ff91b1ebc) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingPolicy.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FifoPolicy.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSLeafQueue.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/DominantResourceFairnessPolicy.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FairSharePolicy.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #244 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/244/)
          YARN-3453. Ensure preemption logic in FairScheduler uses DominantResourceCalculator in DRF queues to prevent unnecessary thrashing. (asuresh) (Arun Suresh: rev ac94ba3e185115b83351e35c610c2b8ff91b1ebc)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/DominantResourceFairnessPolicy.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSLeafQueue.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FifoPolicy.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingPolicy.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FairSharePolicy.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #244 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/244/ ) YARN-3453 . Ensure preemption logic in FairScheduler uses DominantResourceCalculator in DRF queues to prevent unnecessary thrashing. (asuresh) (Arun Suresh: rev ac94ba3e185115b83351e35c610c2b8ff91b1ebc) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/DominantResourceFairnessPolicy.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSLeafQueue.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FifoPolicy.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingPolicy.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FairSharePolicy.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2202 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2202/)
          YARN-3453. Ensure preemption logic in FairScheduler uses DominantResourceCalculator in DRF queues to prevent unnecessary thrashing. (asuresh) (Arun Suresh: rev ac94ba3e185115b83351e35c610c2b8ff91b1ebc)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FifoPolicy.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingPolicy.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/DominantResourceFairnessPolicy.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSLeafQueue.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FairSharePolicy.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2202 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2202/ ) YARN-3453 . Ensure preemption logic in FairScheduler uses DominantResourceCalculator in DRF queues to prevent unnecessary thrashing. (asuresh) (Arun Suresh: rev ac94ba3e185115b83351e35c610c2b8ff91b1ebc) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FifoPolicy.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingPolicy.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/DominantResourceFairnessPolicy.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSLeafQueue.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FairSharePolicy.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #254 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/254/)
          YARN-3453. Ensure preemption logic in FairScheduler uses DominantResourceCalculator in DRF queues to prevent unnecessary thrashing. (asuresh) (Arun Suresh: rev ac94ba3e185115b83351e35c610c2b8ff91b1ebc)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/DominantResourceFairnessPolicy.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FairSharePolicy.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingPolicy.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSLeafQueue.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FifoPolicy.java
          • hadoop-yarn-project/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #254 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/254/ ) YARN-3453 . Ensure preemption logic in FairScheduler uses DominantResourceCalculator in DRF queues to prevent unnecessary thrashing. (asuresh) (Arun Suresh: rev ac94ba3e185115b83351e35c610c2b8ff91b1ebc) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/DominantResourceFairnessPolicy.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FairSharePolicy.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingPolicy.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSLeafQueue.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FifoPolicy.java hadoop-yarn-project/CHANGES.txt

            People

            • Assignee:
              asuresh Arun Suresh
              Reporter:
              ashwinshankar77 Ashwin Shankar
            • Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development