Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-3547

FairScheduler: Apps that have no resource demand should not participate scheduling

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: fairscheduler
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      At present, all of the 'running' apps participate the scheduling process, however, most of them may have no resource demand on a production cluster, as the app's status is running other than waiting for resource at the most of the app's lifetime. It's not a wise way we sort all the 'running' apps and try to fulfill them, especially on a large-scale cluster which has heavy scheduling load.

      1. YARN-3547.001.patch
        6 kB
        Xianyin Xin
      2. YARN-3547.002.patch
        5 kB
        Xianyin Xin
      3. YARN-3547.003.patch
        3 kB
        Xianyin Xin
      4. YARN-3547.004.patch
        2 kB
        Xianyin Xin
      5. YARN-3547.005.patch
        4 kB
        Xianyin Xin
      6. YARN-3547.006.patch
        3 kB
        Xianyin Xin

        Issue Links

          Activity

          Hide
          xinxianyin Xianyin Xin added a comment -

          A suggested solution provided.

          Show
          xinxianyin Xianyin Xin added a comment - A suggested solution provided.
          Hide
          jlowe Jason Lowe added a comment -

          This looks like a duplicate of YARN-2176. There is some discussion in that JIRA on how to approach implementing it for both the capacity scheduler and fair scheduler. At first glance I'm not sure the proposed patch in this JIRA would work, as I believe requests can be present with #containers=0 which would foil the sched.getAppSchedulingInfo().getAllResourceRequests().size() != 0 check.

          Show
          jlowe Jason Lowe added a comment - This looks like a duplicate of YARN-2176 . There is some discussion in that JIRA on how to approach implementing it for both the capacity scheduler and fair scheduler. At first glance I'm not sure the proposed patch in this JIRA would work, as I believe requests can be present with #containers=0 which would foil the sched.getAppSchedulingInfo().getAllResourceRequests().size() != 0 check.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Xianyin Xin, it seems getAllResourceRequests cannot solve the problem, off-switch resource request will be kept in the requests map even if there's no requests. Instead, you can use SchedulerApplicationAttempt.getAppResourceUsage.getPending > 0 to check if the app has "real" requests.

          Show
          leftnoteasy Wangda Tan added a comment - Xianyin Xin , it seems getAllResourceRequests cannot solve the problem, off-switch resource request will be kept in the requests map even if there's no requests. Instead, you can use SchedulerApplicationAttempt.getAppResourceUsage.getPending > 0 to check if the app has "real" requests.
          Hide
          xinxianyin Xianyin Xin added a comment -

          Thanks for pointing out the duplication, Jason Lowe, and thanks for comment, Jason Lowe, Wangda Tan.
          Wangda Tan, are we going to implement non-exclusive node labels support in FairScheduler or is there any plan? If so, this problem can be solved incidentally just like YARN-2176.

          Show
          xinxianyin Xianyin Xin added a comment - Thanks for pointing out the duplication, Jason Lowe , and thanks for comment, Jason Lowe , Wangda Tan . Wangda Tan , are we going to implement non-exclusive node labels support in FairScheduler or is there any plan? If so, this problem can be solved incidentally just like YARN-2176 .
          Hide
          xinxianyin Xianyin Xin added a comment -

          Switch to using SchedulerApplicationAttempt.getAppAttemptResourceUsage.getPending, and change the type of pendingApps from TreeSet to ArrayList.

          Show
          xinxianyin Xianyin Xin added a comment - Switch to using SchedulerApplicationAttempt.getAppAttemptResourceUsage.getPending, and change the type of pendingApps from TreeSet to ArrayList.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Xianyin Xin, just added you to contributor list so that you can assign JIRAs to yourself.

          Looked at your patch, now it make the "pendingApps" to be apps participate scheduling. It's an ArrayList so every add/remove takes O. I think it's better to make it to be a Set to do faster add/remove.

          Show
          leftnoteasy Wangda Tan added a comment - Xianyin Xin , just added you to contributor list so that you can assign JIRAs to yourself. Looked at your patch, now it make the "pendingApps" to be apps participate scheduling. It's an ArrayList so every add/remove takes O . I think it's better to make it to be a Set to do faster add/remove.
          Hide
          xinxianyin Xianyin Xin added a comment -

          Thanks Wangda Tan!
          Attached a new patch using Set. When i review the patch, I found patch-001 is ill-considered, because unlike CS, Fair uses list to organize the apps at the very beginning, thus the newly added sorted set may become un-sorted because of the "update resource usage" or "update fair share" operations. We have two choices, the first is we continue to use the solution in patch-001, but fix it with considerations that we reorder the set when we update resource usage or fair shares just like in CS; the second is like patch-003, before we do assign, we scan the runningApps, and pick out the apps with real request to construct a set. Patch-003 is more efficient than patch-002, and the change of code is less than patch-001 (if fixed). What do you think, Wangda Tan?
          Change the name "pendingApps" to "pendingForResourceApps" because "pending" is ambiguous. Maybe we need a better name.

          Show
          xinxianyin Xianyin Xin added a comment - Thanks Wangda Tan ! Attached a new patch using Set. When i review the patch, I found patch-001 is ill-considered, because unlike CS, Fair uses list to organize the apps at the very beginning, thus the newly added sorted set may become un-sorted because of the "update resource usage" or "update fair share" operations. We have two choices, the first is we continue to use the solution in patch-001, but fix it with considerations that we reorder the set when we update resource usage or fair shares just like in CS; the second is like patch-003, before we do assign, we scan the runningApps, and pick out the apps with real request to construct a set. Patch-003 is more efficient than patch-002, and the change of code is less than patch-001 (if fixed). What do you think, Wangda Tan ? Change the name "pendingApps" to "pendingForResourceApps" because "pending" is ambiguous. Maybe we need a better name.
          Hide
          xinxianyin Xianyin Xin added a comment -

          An alternative solution, scan and pick apps before do scheduling.

          Show
          xinxianyin Xianyin Xin added a comment - An alternative solution, scan and pick apps before do scheduling.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Xianyin Xin,
          Thanks for updating, I think using either TreeSet or List (needs resort) should be fine.

          +Karthik Kambatla.

          Show
          leftnoteasy Wangda Tan added a comment - Xianyin Xin , Thanks for updating, I think using either TreeSet or List (needs resort) should be fine. + Karthik Kambatla .
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 39s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 javac 7m 35s There were no new javac warning messages.
          +1 javadoc 9m 36s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 0m 49s The applied patch generated 1 new checkstyle issues (total was 9, now 10).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 34s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 1m 15s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          -1 yarn tests 52m 58s Tests failed in hadoop-yarn-server-resourcemanager.
              89m 29s  



          Reason Tests
          Failed unit tests hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12730098/YARN-3547.003.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 551615f
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/7696/artifact/patchprocess/diffcheckstylehadoop-yarn-server-resourcemanager.txt
          hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/7696/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7696/testReport/
          Java 1.7.0_55
          uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/7696/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 39s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 35s There were no new javac warning messages. +1 javadoc 9m 36s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 0m 49s The applied patch generated 1 new checkstyle issues (total was 9, now 10). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 1m 15s The patch does not introduce any new Findbugs (version 2.0.3) warnings. -1 yarn tests 52m 58s Tests failed in hadoop-yarn-server-resourcemanager.     89m 29s   Reason Tests Failed unit tests hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12730098/YARN-3547.003.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 551615f checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/7696/artifact/patchprocess/diffcheckstylehadoop-yarn-server-resourcemanager.txt hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/7696/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7696/testReport/ Java 1.7.0_55 uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/7696/console This message was automatically generated.
          Hide
          xinxianyin Xianyin Xin added a comment -

          Fixed test failures. New test is not need.

          Show
          xinxianyin Xianyin Xin added a comment - Fixed test failures. New test is not need.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 41s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 javac 7m 35s There were no new javac warning messages.
          +1 javadoc 9m 33s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 45s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 34s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 1m 17s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          +1 yarn tests 53m 39s Tests passed in hadoop-yarn-server-resourcemanager.
              90m 2s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12730699/YARN-3547.004.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 90b3845
          hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/7724/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7724/testReport/
          Java 1.7.0_55
          uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/7724/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 41s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 35s There were no new javac warning messages. +1 javadoc 9m 33s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 45s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 1m 17s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 yarn tests 53m 39s Tests passed in hadoop-yarn-server-resourcemanager.     90m 2s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12730699/YARN-3547.004.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 90b3845 hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/7724/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7724/testReport/ Java 1.7.0_55 uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/7724/console This message was automatically generated.
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Wangda Tan / Xianyin Xin, can we not use SchedulerApplicationAttempt.hasPendingResourceRequest() the way CS's LeafQueue uses it?

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Wangda Tan / Xianyin Xin , can we not use SchedulerApplicationAttempt.hasPendingResourceRequest() the way CS's LeafQueue uses it?
          Hide
          xinxianyin Xianyin Xin added a comment -

          Thanks for your feedback Vinod Kumar Vavilapalli. We use SchedulerApplicationAttempt.getAppAttemptResourceUsage().getPending(), not hasPendingResourceRequest(). In fact i don't quite understand what you mean. attemptResourceUsage records the resource information of an attempt, no matter it is scheduled by CS or Fair. Would you please explain your concern?

          Show
          xinxianyin Xianyin Xin added a comment - Thanks for your feedback Vinod Kumar Vavilapalli . We use SchedulerApplicationAttempt.getAppAttemptResourceUsage().getPending(), not hasPendingResourceRequest(). In fact i don't quite understand what you mean. attemptResourceUsage records the resource information of an attempt, no matter it is scheduled by CS or Fair. Would you please explain your concern?
          Hide
          kasha Karthik Kambatla added a comment -

          In FSAppAttempt, pending resources can be derived from getDemand() - getResourceUsage(). Can we use that please?

          Show
          kasha Karthik Kambatla added a comment - In FSAppAttempt, pending resources can be derived from getDemand() - getResourceUsage() . Can we use that please?
          Hide
          kasha Karthik Kambatla added a comment -

          Canceling patch to address review feedback.

          Show
          kasha Karthik Kambatla added a comment - Canceling patch to address review feedback.
          Hide
          xinxianyin Xianyin Xin added a comment -

          Sorry for late response, Karthik Kambatla. getDemand() - getResourceUsage() is another choice, but i think SchedulerApplicationAttempt.getAppAttemptResourceUsage().getPending() would be better, because the pending information is immediately updated as the request is update/fulfilled, while the former was derived from request in another update thread. The request is changed when there's resource request update. If AM submits resource requests

          location=*     containers=1
          location=rack  containers=1
          location=node1 containers=1
          location=node2 containers=1
          

          and then it is fulfilled on node1, the request becomes

          location=*     containers=0
          location=node2 containers=1
          

          Later the AM doesn't submit a new request (in which the #container set to 0), this app will also participate scheduling, even though it's demand has been fulfilled.

          Besides, when i try use getDemand() - getResourceUsage(), i got some test failures. Now i was checking the cause.

          Show
          xinxianyin Xianyin Xin added a comment - Sorry for late response, Karthik Kambatla . getDemand() - getResourceUsage() is another choice, but i think SchedulerApplicationAttempt.getAppAttemptResourceUsage().getPending() would be better, because the pending information is immediately updated as the request is update/fulfilled, while the former was derived from request in another update thread. The request is changed when there's resource request update. If AM submits resource requests location=* containers=1 location=rack containers=1 location=node1 containers=1 location=node2 containers=1 and then it is fulfilled on node1, the request becomes location=* containers=0 location=node2 containers=1 Later the AM doesn't submit a new request (in which the #container set to 0), this app will also participate scheduling, even though it's demand has been fulfilled. Besides, when i try use getDemand() - getResourceUsage() , i got some test failures. Now i was checking the cause.
          Hide
          xinxianyin Xianyin Xin added a comment -

          A patch using getDemand() - getResourceUsage().

          Show
          xinxianyin Xianyin Xin added a comment - A patch using getDemand() - getResourceUsage() .
          Hide
          xinxianyin Xianyin Xin added a comment -

          Karthik Kambatla, can you please have a look?

          Show
          xinxianyin Xianyin Xin added a comment - Karthik Kambatla , can you please have a look?
          Hide
          leftnoteasy Wangda Tan added a comment -

          SchedulerApplicationAttempt.getAppAttemptResourceUsage().getPending() is always up-to-dated, FS can use it to do pending resource updating.

          Show
          leftnoteasy Wangda Tan added a comment - SchedulerApplicationAttempt.getAppAttemptResourceUsage().getPending() is always up-to-dated, FS can use it to do pending resource updating.
          Hide
          xinxianyin Xianyin Xin added a comment -

          Agree, Wangda Tan. Now we have YARN-3547.004.patch using SchedulerApplicationAttempt.getAppAttemptResourceUsage().getPending() and YARN-3547.005.patch using getDemand() - getResourceUsage().

          Show
          xinxianyin Xianyin Xin added a comment - Agree, Wangda Tan . Now we have YARN-3547 .004.patch using SchedulerApplicationAttempt.getAppAttemptResourceUsage().getPending() and YARN-3547 .005.patch using getDemand() - getResourceUsage() .
          Hide
          xinxianyin Xianyin Xin added a comment -

          Hi Karthik Kambatla, Wangda Tan, can we reach a consensus as the patch is just a simple fix?

          Show
          xinxianyin Xianyin Xin added a comment - Hi Karthik Kambatla , Wangda Tan , can we reach a consensus as the patch is just a simple fix?
          Hide
          leftnoteasy Wangda Tan added a comment -

          Karthik Kambatla, do you have any concern if using SchedulerApplicationAttempt.getAppAttemptResourceUsage().getPending().

          Show
          leftnoteasy Wangda Tan added a comment - Karthik Kambatla , do you have any concern if using SchedulerApplicationAttempt.getAppAttemptResourceUsage().getPending() .
          Hide
          kasha Karthik Kambatla added a comment -

          Sorry for the delay in response here. I am okay with using SchedulerApplicationAttempt.getAppAttemptResourceUsage().getPending(). I don't anticipate any issues due to potential mismatch between this value and getDemand - getResourceUsage. We can always fix them if we run into them.

          Show
          kasha Karthik Kambatla added a comment - Sorry for the delay in response here. I am okay with using SchedulerApplicationAttempt.getAppAttemptResourceUsage().getPending(). I don't anticipate any issues due to potential mismatch between this value and getDemand - getResourceUsage. We can always fix them if we run into them.
          Hide
          leftnoteasy Wangda Tan added a comment -

          I'm fine with the approach suggested by Karthik Kambatla. Xianyin Xin, you can figure out with Karthik about how to continue with the patch.

          Show
          leftnoteasy Wangda Tan added a comment - I'm fine with the approach suggested by Karthik Kambatla . Xianyin Xin , you can figure out with Karthik about how to continue with the patch.
          Hide
          xinxianyin Xianyin Xin added a comment -

          Thanks Karthik Kambatla, Wangda Tan.
          Karthik Kambatla, do you have any other concerns on YARN-3547.005.patch?

          Show
          xinxianyin Xianyin Xin added a comment - Thanks Karthik Kambatla , Wangda Tan . Karthik Kambatla , do you have any other concerns on YARN-3547 .005.patch?
          Hide
          kasha Karthik Kambatla added a comment -

          Per recent discussion, let us go with the approach in YARN-3547.004.patch?

          Review comments for 004 patch:

          1. What is the readLock protecting? If it is only runnableApps, we should release the lock as soon as we are done iterating through runnableApps.
          2. Use (!pending.equals(Resources.none()) instead of an elaborate check?
          3. We can avoid importing Set by just using TreeSet. Since it is all local, that should be fine.
          Show
          kasha Karthik Kambatla added a comment - Per recent discussion, let us go with the approach in YARN-3547 .004.patch? Review comments for 004 patch: What is the readLock protecting? If it is only runnableApps, we should release the lock as soon as we are done iterating through runnableApps. Use (!pending.equals(Resources.none()) instead of an elaborate check? We can avoid importing Set by just using TreeSet . Since it is all local, that should be fine.
          Hide
          xinxianyin Xianyin Xin added a comment -

          Sorry Karthik Kambatla, i misunderstood the discussion. Anyway, thanks for your reviewing.
          Only comment on point 2:
          In fact i firstly use (!pending.equals(Resources.none()), however, a test case in TestFairScheduler is tesing zero memory container request <0MB, 1core>. To avoid this, i changed the check. But it seems the test has been removed recently.
          Now all of the 3 points are addressed.

          Show
          xinxianyin Xianyin Xin added a comment - Sorry Karthik Kambatla , i misunderstood the discussion. Anyway, thanks for your reviewing. Only comment on point 2: In fact i firstly use (!pending.equals(Resources.none()) , however, a test case in TestFairScheduler is tesing zero memory container request <0MB, 1core>. To avoid this, i changed the check. But it seems the test has been removed recently. Now all of the 3 points are addressed.
          Hide
          xinxianyin Xianyin Xin added a comment -

          Upload YARN-3547.006.patch.

          Show
          xinxianyin Xianyin Xin added a comment - Upload YARN-3547 .006.patch.
          Hide
          xinxianyin Xianyin Xin added a comment -

          Sorry Karthik Kambatla, i correct the comment in https://issues.apache.org/jira/browse/YARN-3547?focusedCommentId=14564204&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14564204. I firstly use Resources.greaterThan(Resources.none()), which is not suitable for zero memory container request. But (!pending.equals(Resources.none()) can.

          Show
          xinxianyin Xianyin Xin added a comment - Sorry Karthik Kambatla , i correct the comment in https://issues.apache.org/jira/browse/YARN-3547?focusedCommentId=14564204&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14564204 . I firstly use Resources.greaterThan(Resources.none()), which is not suitable for zero memory container request. But (!pending.equals(Resources.none()) can.
          Hide
          kasha Karthik Kambatla added a comment -

          +1, pending Jenkins.

          At commit time, I plan to add the following comment: "Note that we might consider a non-runnable app for scheduling if it was just removed from runnableApps. This should be okay in practice - (1) because that is rare, and (2) holding the lock to disallow that doesn't alter the reality where conditions for an app to be made non-runnable have already been met."

          Show
          kasha Karthik Kambatla added a comment - +1, pending Jenkins. At commit time, I plan to add the following comment: "Note that we might consider a non-runnable app for scheduling if it was just removed from runnableApps. This should be okay in practice - (1) because that is rare, and (2) holding the lock to disallow that doesn't alter the reality where conditions for an app to be made non-runnable have already been met."
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 16m 8s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 javac 7m 41s There were no new javac warning messages.
          +1 javadoc 9m 44s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 46s There were no new checkstyle issues.
          +1 whitespace 0m 1s The patch has no lines that end in whitespace.
          +1 install 1m 34s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 1m 26s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 50m 20s Tests passed in hadoop-yarn-server-resourcemanager.
              88m 39s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12736049/YARN-3547.006.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 7817674
          hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8130/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8130/testReport/
          Java 1.7.0_55
          uname Linux asf908.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8130/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 8s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 41s There were no new javac warning messages. +1 javadoc 9m 44s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 46s There were no new checkstyle issues. +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 1m 26s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 50m 20s Tests passed in hadoop-yarn-server-resourcemanager.     88m 39s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12736049/YARN-3547.006.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 7817674 hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8130/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8130/testReport/ Java 1.7.0_55 uname Linux asf908.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8130/console This message was automatically generated.
          Hide
          kasha Karthik Kambatla added a comment -

          Thanks for reporting and working on this, Xianyin. Just committed this to trunk and branch-2.

          Show
          kasha Karthik Kambatla added a comment - Thanks for reporting and working on this, Xianyin. Just committed this to trunk and branch-2.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #7928 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7928/)
          YARN-3547. FairScheduler: Apps that have no resource demand should not participate scheduling. (Xianyin Xin via kasha) (kasha: rev 3ae2a625018bc8cf431aa19da5bf8fe4ef8c1ad4)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java
          • hadoop-yarn-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7928 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7928/ ) YARN-3547 . FairScheduler: Apps that have no resource demand should not participate scheduling. (Xianyin Xin via kasha) (kasha: rev 3ae2a625018bc8cf431aa19da5bf8fe4ef8c1ad4) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java hadoop-yarn-project/CHANGES.txt
          Hide
          xinxianyin Xianyin Xin added a comment -

          Thanks Karthik Kambatla!
          Thanks Karthik Kambatla, Wangda Tan and Vinod Kumar Vavilapalli for your valuable comments!

          Show
          xinxianyin Xianyin Xin added a comment - Thanks Karthik Kambatla ! Thanks Karthik Kambatla , Wangda Tan and Vinod Kumar Vavilapalli for your valuable comments!
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #943 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/943/)
          YARN-3547. FairScheduler: Apps that have no resource demand should not participate scheduling. (Xianyin Xin via kasha) (kasha: rev 3ae2a625018bc8cf431aa19da5bf8fe4ef8c1ad4)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java
          • hadoop-yarn-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #943 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/943/ ) YARN-3547 . FairScheduler: Apps that have no resource demand should not participate scheduling. (Xianyin Xin via kasha) (kasha: rev 3ae2a625018bc8cf431aa19da5bf8fe4ef8c1ad4) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java hadoop-yarn-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #213 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/213/)
          YARN-3547. FairScheduler: Apps that have no resource demand should not participate scheduling. (Xianyin Xin via kasha) (kasha: rev 3ae2a625018bc8cf431aa19da5bf8fe4ef8c1ad4)

          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #213 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/213/ ) YARN-3547 . FairScheduler: Apps that have no resource demand should not participate scheduling. (Xianyin Xin via kasha) (kasha: rev 3ae2a625018bc8cf431aa19da5bf8fe4ef8c1ad4) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Hdfs-trunk #2141 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2141/)
          YARN-3547. FairScheduler: Apps that have no resource demand should not participate scheduling. (Xianyin Xin via kasha) (kasha: rev 3ae2a625018bc8cf431aa19da5bf8fe4ef8c1ad4)

          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #2141 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2141/ ) YARN-3547 . FairScheduler: Apps that have no resource demand should not participate scheduling. (Xianyin Xin via kasha) (kasha: rev 3ae2a625018bc8cf431aa19da5bf8fe4ef8c1ad4) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #202 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/202/)
          YARN-3547. FairScheduler: Apps that have no resource demand should not participate scheduling. (Xianyin Xin via kasha) (kasha: rev 3ae2a625018bc8cf431aa19da5bf8fe4ef8c1ad4)

          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #202 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/202/ ) YARN-3547 . FairScheduler: Apps that have no resource demand should not participate scheduling. (Xianyin Xin via kasha) (kasha: rev 3ae2a625018bc8cf431aa19da5bf8fe4ef8c1ad4) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #211 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/211/)
          YARN-3547. FairScheduler: Apps that have no resource demand should not participate scheduling. (Xianyin Xin via kasha) (kasha: rev 3ae2a625018bc8cf431aa19da5bf8fe4ef8c1ad4)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java
          • hadoop-yarn-project/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #211 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/211/ ) YARN-3547 . FairScheduler: Apps that have no resource demand should not participate scheduling. (Xianyin Xin via kasha) (kasha: rev 3ae2a625018bc8cf431aa19da5bf8fe4ef8c1ad4) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java hadoop-yarn-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2159 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2159/)
          YARN-3547. FairScheduler: Apps that have no resource demand should not participate scheduling. (Xianyin Xin via kasha) (kasha: rev 3ae2a625018bc8cf431aa19da5bf8fe4ef8c1ad4)

          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2159 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2159/ ) YARN-3547 . FairScheduler: Apps that have no resource demand should not participate scheduling. (Xianyin Xin via kasha) (kasha: rev 3ae2a625018bc8cf431aa19da5bf8fe4ef8c1ad4) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java

            People

            • Assignee:
              xinxianyin Xianyin Xin
              Reporter:
              xinxianyin Xianyin Xin
            • Votes:
              0 Vote for this issue
              Watchers:
              16 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development