Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-3485

FairScheduler headroom calculation doesn't consider maxResources for Fifo and FairShare policies

        Details

        • Target Version/s:
        • Hadoop Flags:
          Reviewed

          Description

          FairScheduler's headroom calculations consider the fairshare and cluster-available-resources, and the fairshare has maxResources. However, for Fifo and Fairshare policies, the fairshare is used only for memory and not cpu. So, the scheduler ends up showing a higher headroom than is actually available. This could lead to applications waiting for resources far longer than then intend to. e.g. MAPREDUCE-6302.

          1. Text File
            yarn-3485-1.patch
            13 kB
            Karthik Kambatla
          2. Text File
            yarn-3485-2.patch
            13 kB
            Karthik Kambatla
          3. Text File
            yarn-3485-3.patch
            13 kB
            Karthik Kambatla
          4. Text File
            yarn-3485-prelim.patch
            8 kB
            Karthik Kambatla

            Issue Links

            is duplicated by

            Bug - A problem which impairs or prevents the functions of the product. YARN-3891 Apps are starved for resource due to default FairSharePolicy in Fair Scheduler

            • Critical - Crashes, loss of data, severe memory leak.
            • Resolved
            relates to

            Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-6302 Preempt reducers after a configurable timeout irrespective of headroom

            • Critical - Crashes, loss of data, severe memory leak.
            • Closed

              Activity

              Ascending order - Click to sort in descending order
              Hide
              Permalink
              kasha Karthik Kambatla added a comment -

              Preliminary patch that considers maxShare. Appreciate any early reviews.

              Show
              kasha Karthik Kambatla added a comment - Preliminary patch that considers maxShare. Appreciate any early reviews.
              Hide
              Permalink
              adhoot Anubhav Dhoot added a comment -

              Looks like a good change. This makes the headroom more accurate as its taking into account queue maxshare.

              Show
              adhoot Anubhav Dhoot added a comment - Looks like a good change. This makes the headroom more accurate as its taking into account queue maxshare.
              Hide
              Permalink
              peng.zhang Peng Zhang added a comment - 
              +    maxAvailableResource.setMemory(
+        Math.min(maxAvailableResource.getMemory(),
+            queue.getMaxShare().getMemory()));
+    maxAvailableResource.setVirtualCores(
+        Math.min(maxAvailableResource.getVirtualCores(),
+            queue.getMaxShare().getVirtualCores()));

              use Resources.componentwiseMin() better

              Show
              peng.zhang Peng Zhang added a comment - + maxAvailableResource.setMemory( + Math .min(maxAvailableResource.getMemory(), + queue.getMaxShare().getMemory())); + maxAvailableResource.setVirtualCores( + Math .min(maxAvailableResource.getVirtualCores(), + queue.getMaxShare().getVirtualCores())); use Resources.componentwiseMin() better
              Hide
              Permalink
              peng.zhang Peng Zhang added a comment -

              some thoughts for this issue:

              1. I think for fair and fifo, queue's fair share should already be 0, so no need not to use min(maxAvailableResource, queue.getMaxShare()) in first. And normally, for non-drf policy, max memory usually is set arbitrarily without real meaning.
              2. I think even if headroom of fair and fifo maybe got wrong value in cpu dimension, MapReduce will not request wrong number of map and reduces. Because MapReduce use ResourceCalculatorUtils.computeAvailableContainers() to compute how much containers to ask. It will use min containers computed by two dimensions. As MAPREDUCE-6302, I think if it's related with headroom, it is probably wrong in both dimensions.
              Show
              peng.zhang Peng Zhang added a comment - some thoughts for this issue: I think for fair and fifo, queue's fair share should already be 0, so no need not to use min(maxAvailableResource, queue.getMaxShare()) in first. And normally, for non-drf policy, max memory usually is set arbitrarily without real meaning. I think even if headroom of fair and fifo maybe got wrong value in cpu dimension, MapReduce will not request wrong number of map and reduces. Because MapReduce use ResourceCalculatorUtils.computeAvailableContainers() to compute how much containers to ask. It will use min containers computed by two dimensions. As MAPREDUCE-6302 , I think if it's related with headroom, it is probably wrong in both dimensions.
              Hide
              Permalink
              peng.zhang Peng Zhang added a comment -

              max memory cpu usually is set arbitrarily without real meaning.

              Show
              peng.zhang Peng Zhang added a comment - max memory cpu usually is set arbitrarily without real meaning.
              Hide
              Permalink
              kasha Karthik Kambatla added a comment -

              Peng Zhang - I am not sure I understand your latest comments.

              FairScheduler's Fifo and Fair policies don't consider CPU max resources when calculating headroom. However, these policies consider CPU at allocation time. So, the potential maximum allocation for an app ends up being smaller than the headroom. If apps wait for headroom amount of resources, they will never get them. This JIRA attempts to fix that issue.

              Show
              kasha Karthik Kambatla added a comment - Peng Zhang - I am not sure I understand your latest comments. FairScheduler's Fifo and Fair policies don't consider CPU max resources when calculating headroom. However, these policies consider CPU at allocation time. So, the potential maximum allocation for an app ends up being smaller than the headroom. If apps wait for headroom amount of resources, they will never get them. This JIRA attempts to fix that issue.
              Hide
              Permalink
              kasha Karthik Kambatla added a comment -

              Updated the patch to use Resources.componentWiseMin and add unit test to verify the fix.

              Show
              kasha Karthik Kambatla added a comment - Updated the patch to use Resources.componentWiseMin and add unit test to verify the fix.
              Hide
              Permalink
              kasha Karthik Kambatla added a comment -

              Minor fix up in the test.

              Show
              kasha Karthik Kambatla added a comment - Minor fix up in the test.
              Hide
              Permalink
              hadoopqa Hadoop QA added a comment -



              -1 overall



              Vote Subsystem Runtime Comment
              0 pre-patch 14m 47s Pre-patch trunk compilation is healthy.
              +1 @author 0m 0s The patch does not contain any @author tags.
              +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
              +1 whitespace 0m 0s The patch has no lines that end in whitespace.
              +1 javac 7m 32s There were no new javac warning messages.
              +1 javadoc 9m 38s There were no new javadoc warning messages.
              +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
              +1 checkstyle 5m 25s There were no new checkstyle issues.
              +1 install 1m 34s mvn install still works.
              +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
              +1 findbugs 1m 15s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
              -1 yarn tests 52m 18s Tests failed in hadoop-yarn-server-resourcemanager.
                  93m 27s  



              Reason Tests
              Failed unit tests hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart



              Subsystem Report/Notes
              Patch URL http://issues.apache.org/jira/secure/attachment/12728115/yarn-3485-1.patch
              Optional Tests javadoc javac unit findbugs checkstyle
              git revision trunk / 78c6b46
              hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/7499/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
              Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7499/testReport/
              Console output https://builds.apache.org/job/PreCommit-YARN-Build/7499/console

              This message was automatically generated.

              Show
              hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 47s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 javac 7m 32s There were no new javac warning messages. +1 javadoc 9m 38s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 5m 25s There were no new checkstyle issues. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 1m 15s The patch does not introduce any new Findbugs (version 2.0.3) warnings. -1 yarn tests 52m 18s Tests failed in hadoop-yarn-server-resourcemanager.     93m 27s   Reason Tests Failed unit tests hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12728115/yarn-3485-1.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 78c6b46 hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/7499/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7499/testReport/ Console output https://builds.apache.org/job/PreCommit-YARN-Build/7499/console This message was automatically generated.
              Hide
              Permalink
              kasha Karthik Kambatla added a comment -

              The test failure is unrelated.

              Show
              kasha Karthik Kambatla added a comment - The test failure is unrelated.
              Hide
              Permalink
              asuresh Arun Suresh added a comment -

              +1, LGTM
              Thanks for working on this Karthik Kambatla

              Show
              asuresh Arun Suresh added a comment - +1, LGTM Thanks for working on this Karthik Kambatla
              Hide
              Permalink
              sandyr Sandy Ryza added a comment -

              It looks like the patch computes the headroom as min(cluster total - cluster consumed, queue max resource). Do we not want it to be min(cluster total - cluster consumed, queue max resource - queue consumed)?

              Show
              sandyr Sandy Ryza added a comment - It looks like the patch computes the headroom as min(cluster total - cluster consumed, queue max resource). Do we not want it to be min(cluster total - cluster consumed, queue max resource - queue consumed)?
              Hide
              Permalink
              kasha Karthik Kambatla added a comment -

              Thanks for the review, Sandy. Good catch! Updated patch to use queue's available maxResources instead.

              Show
              kasha Karthik Kambatla added a comment - Thanks for the review, Sandy. Good catch! Updated patch to use queue's available maxResources instead.
              Hide
              Permalink
              sandyr Sandy Ryza added a comment -

              One nit:

              +    return Math.min( Math.min(value1, value2), value3);

              has an extra space.

              Otherwise +1.

              Show
              sandyr Sandy Ryza added a comment - One nit: + return Math .min( Math .min(value1, value2), value3); has an extra space. Otherwise +1.
              Hide
              Permalink
              hadoopqa Hadoop QA added a comment -



              +1 overall



              Vote Subsystem Runtime Comment
              0 pre-patch 14m 39s Pre-patch trunk compilation is healthy.
              +1 @author 0m 0s The patch does not contain any @author tags.
              +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
              +1 whitespace 0m 1s The patch has no lines that end in whitespace.
              +1 javac 7m 32s There were no new javac warning messages.
              +1 javadoc 9m 33s There were no new javadoc warning messages.
              +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
              +1 checkstyle 7m 33s There were no new checkstyle issues.
              +1 install 1m 35s mvn install still works.
              +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
              +1 findbugs 1m 17s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
              +1 yarn tests 52m 7s Tests passed in hadoop-yarn-server-resourcemanager.
                  95m 15s  



              Subsystem Report/Notes
              Patch URL http://issues.apache.org/jira/secure/attachment/12728948/yarn-3485-2.patch
              Optional Tests javadoc javac unit findbugs checkstyle
              git revision trunk / 5190923
              hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/7529/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
              Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7529/testReport/
              Java 1.7.0_55
              uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
              Console output https://builds.apache.org/job/PreCommit-YARN-Build/7529/console

              This message was automatically generated.

              Show
              hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 39s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 javac 7m 32s There were no new javac warning messages. +1 javadoc 9m 33s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 7m 33s There were no new checkstyle issues. +1 install 1m 35s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 1m 17s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 yarn tests 52m 7s Tests passed in hadoop-yarn-server-resourcemanager.     95m 15s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12728948/yarn-3485-2.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 5190923 hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/7529/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7529/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/7529/console This message was automatically generated.
              Hide
              Permalink
              kasha Karthik Kambatla added a comment -

              Thanks for the review, Sandy. Uploaded patch addresses the nit.

              Show
              kasha Karthik Kambatla added a comment - Thanks for the review, Sandy. Uploaded patch addresses the nit.
              Hide
              Permalink
              kasha Karthik Kambatla added a comment -

              Committing this without waiting for a Jenkins run for the white-space change.

              Show
              kasha Karthik Kambatla added a comment - Committing this without waiting for a Jenkins run for the white-space change.
              Hide
              Permalink
              kasha Karthik Kambatla added a comment -

              Committed to trunk, branch-2 and branch-2.7 (after running TestFSAppAttempt for sanity).

              Thanks Sandy, Anubhav and Arun for the reviews.

              Show
              kasha Karthik Kambatla added a comment - Committed to trunk, branch-2 and branch-2.7 (after running TestFSAppAttempt for sanity). Thanks Sandy, Anubhav and Arun for the reviews.
              Hide
              Permalink
              hudson Hudson added a comment -

              FAILURE: Integrated in Hadoop-trunk-Commit #7698 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7698/)
              YARN-3485. FairScheduler headroom calculation doesn't consider maxResources for Fifo and FairShare policies. (kasha) (kasha: rev 8f82970e0c247b37b2bf33aa21f6a39afa07efde)

              • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSAppAttempt.java
              • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FifoPolicy.java
              • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FairSharePolicy.java
              • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java
              • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingPolicy.java
              • hadoop-yarn-project/CHANGES.txt
              • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/DominantResourceFairnessPolicy.java
              Show
              hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7698 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7698/ ) YARN-3485 . FairScheduler headroom calculation doesn't consider maxResources for Fifo and FairShare policies. (kasha) (kasha: rev 8f82970e0c247b37b2bf33aa21f6a39afa07efde) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSAppAttempt.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FifoPolicy.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FairSharePolicy.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingPolicy.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/DominantResourceFairnessPolicy.java
              Hide
              Permalink
              hudson Hudson added a comment -

              FAILURE: Integrated in Hadoop-Hdfs-trunk #2110 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2110/)
              YARN-3485. FairScheduler headroom calculation doesn't consider maxResources for Fifo and FairShare policies. (kasha) (kasha: rev 8f82970e0c247b37b2bf33aa21f6a39afa07efde)

              • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSAppAttempt.java
              • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/DominantResourceFairnessPolicy.java
              • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingPolicy.java
              • hadoop-yarn-project/CHANGES.txt
              • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FairSharePolicy.java
              • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FifoPolicy.java
              • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java
              Show
              hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2110 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2110/ ) YARN-3485 . FairScheduler headroom calculation doesn't consider maxResources for Fifo and FairShare policies. (kasha) (kasha: rev 8f82970e0c247b37b2bf33aa21f6a39afa07efde) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSAppAttempt.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/DominantResourceFairnessPolicy.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingPolicy.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FairSharePolicy.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FifoPolicy.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java
              Hide
              Permalink
              hudson Hudson added a comment -

              FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #169 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/169/)
              YARN-3485. FairScheduler headroom calculation doesn't consider maxResources for Fifo and FairShare policies. (kasha) (kasha: rev 8f82970e0c247b37b2bf33aa21f6a39afa07efde)

              • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/DominantResourceFairnessPolicy.java
              • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FifoPolicy.java
              • hadoop-yarn-project/CHANGES.txt
              • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java
              • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSAppAttempt.java
              • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FairSharePolicy.java
              • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingPolicy.java
              Show
              hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #169 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/169/ ) YARN-3485 . FairScheduler headroom calculation doesn't consider maxResources for Fifo and FairShare policies. (kasha) (kasha: rev 8f82970e0c247b37b2bf33aa21f6a39afa07efde) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/DominantResourceFairnessPolicy.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FifoPolicy.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSAppAttempt.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FairSharePolicy.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingPolicy.java
              Hide
              Permalink
              hudson Hudson added a comment -

              FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #178 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/178/)
              YARN-3485. FairScheduler headroom calculation doesn't consider maxResources for Fifo and FairShare policies. (kasha) (kasha: rev 8f82970e0c247b37b2bf33aa21f6a39afa07efde)

              • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSAppAttempt.java
              • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/DominantResourceFairnessPolicy.java
              • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java
              • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FifoPolicy.java
              • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FairSharePolicy.java
              • hadoop-yarn-project/CHANGES.txt
              • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingPolicy.java
              Show
              hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #178 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/178/ ) YARN-3485 . FairScheduler headroom calculation doesn't consider maxResources for Fifo and FairShare policies. (kasha) (kasha: rev 8f82970e0c247b37b2bf33aa21f6a39afa07efde) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSAppAttempt.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/DominantResourceFairnessPolicy.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FifoPolicy.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FairSharePolicy.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingPolicy.java
              Hide
              Permalink
              hudson Hudson added a comment -

              FAILURE: Integrated in Hadoop-Yarn-trunk #912 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/912/)
              YARN-3485. FairScheduler headroom calculation doesn't consider maxResources for Fifo and FairShare policies. (kasha) (kasha: rev 8f82970e0c247b37b2bf33aa21f6a39afa07efde)

              • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FairSharePolicy.java
              • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/DominantResourceFairnessPolicy.java
              • hadoop-yarn-project/CHANGES.txt
              • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSAppAttempt.java
              • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java
              • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FifoPolicy.java
              • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingPolicy.java
              Show
              hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #912 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/912/ ) YARN-3485 . FairScheduler headroom calculation doesn't consider maxResources for Fifo and FairShare policies. (kasha) (kasha: rev 8f82970e0c247b37b2bf33aa21f6a39afa07efde) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FairSharePolicy.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/DominantResourceFairnessPolicy.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSAppAttempt.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FifoPolicy.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingPolicy.java
              Hide
              Permalink
              hudson Hudson added a comment -

              FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #179 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/179/)
              YARN-3485. FairScheduler headroom calculation doesn't consider maxResources for Fifo and FairShare policies. (kasha) (kasha: rev 8f82970e0c247b37b2bf33aa21f6a39afa07efde)

              • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FairSharePolicy.java
              • hadoop-yarn-project/CHANGES.txt
              • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/DominantResourceFairnessPolicy.java
              • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java
              • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingPolicy.java
              • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSAppAttempt.java
              • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FifoPolicy.java
              Show
              hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #179 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/179/ ) YARN-3485 . FairScheduler headroom calculation doesn't consider maxResources for Fifo and FairShare policies. (kasha) (kasha: rev 8f82970e0c247b37b2bf33aa21f6a39afa07efde) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FairSharePolicy.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/DominantResourceFairnessPolicy.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingPolicy.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSAppAttempt.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FifoPolicy.java
              Hide
              Permalink
              hudson Hudson added a comment -

              FAILURE: Integrated in Hadoop-Mapreduce-trunk #2128 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2128/)
              YARN-3485. FairScheduler headroom calculation doesn't consider maxResources for Fifo and FairShare policies. (kasha) (kasha: rev 8f82970e0c247b37b2bf33aa21f6a39afa07efde)

              • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java
              • hadoop-yarn-project/CHANGES.txt
              • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/DominantResourceFairnessPolicy.java
              • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FairSharePolicy.java
              • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FifoPolicy.java
              • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSAppAttempt.java
              • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingPolicy.java
              Show
              hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2128 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2128/ ) YARN-3485 . FairScheduler headroom calculation doesn't consider maxResources for Fifo and FairShare policies. (kasha) (kasha: rev 8f82970e0c247b37b2bf33aa21f6a39afa07efde) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/DominantResourceFairnessPolicy.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FairSharePolicy.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FifoPolicy.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSAppAttempt.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingPolicy.java
              Hide
              Permalink
              l201514 Siqi Li added a comment -

              The latest patch can be applied to 2.6.0 branch cleanly

              Show
              l201514 Siqi Li added a comment - The latest patch can be applied to 2.6.0 branch cleanly

                People

                • Assignee:
                  kasha Karthik Kambatla
                  Reporter:
                  kasha Karthik Kambatla
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  10 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Development