Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2788

Normalize requests in FifoScheduler.allocate to prevent NPEs later

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.23.0
    • Component/s: mrv2
    • Labels:
      None

      Description

      The assignContainer() method in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue can cause the scheduler to crash if the ResourseRequest capability memory == 0 (divide by zero).

      1. MAPREDUCE-2788.patch
        2 kB
        Ahmed Radwan
      2. MAPREDUCE-2788_rev2.patch
        4 kB
        Ahmed Radwan
      3. MAPREDUCE-2788_rev3.patch
        2 kB
        Ahmed Radwan
      4. MAPREDUCE-2788_rev4.patch
        6 kB
        Ahmed Radwan
      5. MAPREDUCE-2788_rev5.patch
        9 kB
        Ahmed Radwan

        Activity

        Ahmed Radwan created issue -
        Hide
        Ahmed Radwan added a comment -

        The attached patch avoids the described crash and returns Resources.none() in case memory==0.

        Show
        Ahmed Radwan added a comment - The attached patch avoids the described crash and returns Resources.none() in case memory==0.
        Ahmed Radwan made changes -
        Field Original Value New Value
        Attachment MAPREDUCE-2788.patch [ 12489780 ]
        Hide
        Arun C Murthy added a comment -

        Ahmed - another way would be to ensure there are no requests which need memory=0 - this check can be enforced in CS.normalizeRequests. Do you want to take a crack at that? Also, please mark this as PA when done - that way I (or other committers) can pick it up on my PA filter. Thanks.

        Show
        Arun C Murthy added a comment - Ahmed - another way would be to ensure there are no requests which need memory=0 - this check can be enforced in CS.normalizeRequests. Do you want to take a crack at that? Also, please mark this as PA when done - that way I (or other committers) can pick it up on my PA filter. Thanks.
        Hide
        Ahmed Radwan added a comment -

        Thanks Arun, I have updated the patch accordingly. Using CS.normalizeRequest, the request's memory is now set to minMemory if it is zero (avoiding the divide by zero). The revised patch is based on the recent version (after merge to trunk).

        Show
        Ahmed Radwan added a comment - Thanks Arun, I have updated the patch accordingly. Using CS.normalizeRequest, the request's memory is now set to minMemory if it is zero (avoiding the divide by zero). The revised patch is based on the recent version (after merge to trunk).
        Ahmed Radwan made changes -
        Attachment MAPREDUCE-2788_rev2.patch [ 12490902 ]
        Ahmed Radwan made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Arun C Murthy added a comment -

        Ahmed - sorry I wasn't clear. I was thinking along hte lines of ensuring all incoming requests (via CS.allocate) are normalized to ensure LeafQueue doesn't have to worry about normalizing them. Thoughts?

        Show
        Arun C Murthy added a comment - Ahmed - sorry I wasn't clear. I was thinking along hte lines of ensuring all incoming requests (via CS.allocate) are normalized to ensure LeafQueue doesn't have to worry about normalizing them. Thoughts?
        Arun C Murthy made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Hide
        Ahmed Radwan added a comment -

        Thanks Arun! The LeafQueue uses the CapacitySchedulerContext interface (CS is an implementation), so other implementations can be added. We don't need to tie the LeafQueue to the CS implementation. So I think the LeafQueue.assignContainer() should check/ensure that the request has no zero memory request before dividing by zero. In my last patch I added normalizeRequest() to the interface so any other implementations are enforced to implement this method (which takes care of this zero memory case). What do you think?

        Show
        Ahmed Radwan added a comment - Thanks Arun! The LeafQueue uses the CapacitySchedulerContext interface (CS is an implementation), so other implementations can be added. We don't need to tie the LeafQueue to the CS implementation. So I think the LeafQueue.assignContainer() should check/ensure that the request has no zero memory request before dividing by zero. In my last patch I added normalizeRequest() to the interface so any other implementations are enforced to implement this method (which takes care of this zero memory case). What do you think?
        Arun C Murthy made changes -
        Priority Major [ 3 ] Critical [ 2 ]
        Hide
        Arun C Murthy added a comment -

        Ahmed - the LeafQueue is interminably tied to CS and hence doing it in CS.allocate is reasonable.

        Show
        Arun C Murthy added a comment - Ahmed - the LeafQueue is interminably tied to CS and hence doing it in CS.allocate is reasonable.
        Hide
        Ahmed Radwan added a comment -

        Thanks Arun, will take a look.

        Show
        Ahmed Radwan added a comment - Thanks Arun, will take a look.
        Hide
        Arun C Murthy added a comment -

        Thanks Ahmed.

        Show
        Arun C Murthy added a comment - Thanks Ahmed.
        Hide
        Arun C Murthy added a comment -

        Ahmed, do you think you'll get to this soon? Thanks.

        Show
        Arun C Murthy added a comment - Ahmed, do you think you'll get to this soon? Thanks.
        Hide
        Ahmed Radwan added a comment -

        Arun, Yes I am working on it. Planning to post an update later today. Thanks!

        Show
        Ahmed Radwan added a comment - Arun, Yes I am working on it. Planning to post an update later today. Thanks!
        Hide
        Ahmed Radwan added a comment -

        Thanks Arun for your comments, I have looked into normalizing the requests within the CapacityScheduler.

        It doesn't seem that the call to LeafQueue.assignContainer(..) come via CapacityScheduler.allocate(). It gets called through the call path:

        LeafQueue.assignContainer(..) <- assignNodeLocalContainers(..) <-- LeafQueue.assignContainersOnNode(..) <- LeafQueue.assignContainers(..)

        There are alternative paths, but all lead to the same source.

        The SchedulerApp application (in the LeafQueue.assignContainers(..) call) is one of the Map<ApplicationAttemptId, SchedulerApp> applicationsMap values. This applicationsMap is only populated through LeafQueue.addApplication(..).

        The LeafQueue.addApplication(..) is called through the path: LeafQueue.addApplication(..) <- LeafQueue.submitApplication(..) <- CapacityScheduler.addApplication(..).

        So I have added code to CapacityScheduler.addApplication(..) to normalize all resource requests for the SchedulerApp before submitting to the queue.

        If the LeafQueue is interminably tied to CS, we may need to update the references in LeafQueue to use CapacityScheduler instead of CapacitySchedulerContext, this will make such dependency clear and avoid future confusions. I haven't made this interface change in the attached patch, as it requires more changes to other components, but if we agree about it, I can do it in a following issue.

        Show
        Ahmed Radwan added a comment - Thanks Arun for your comments, I have looked into normalizing the requests within the CapacityScheduler. It doesn't seem that the call to LeafQueue.assignContainer(..) come via CapacityScheduler.allocate(). It gets called through the call path: LeafQueue.assignContainer(..) <- assignNodeLocalContainers(..) <-- LeafQueue.assignContainersOnNode(..) <- LeafQueue.assignContainers(..) There are alternative paths, but all lead to the same source. The SchedulerApp application (in the LeafQueue.assignContainers(..) call) is one of the Map<ApplicationAttemptId, SchedulerApp> applicationsMap values. This applicationsMap is only populated through LeafQueue.addApplication(..). The LeafQueue.addApplication(..) is called through the path: LeafQueue.addApplication(..) <- LeafQueue.submitApplication(..) <- CapacityScheduler.addApplication(..). So I have added code to CapacityScheduler.addApplication(..) to normalize all resource requests for the SchedulerApp before submitting to the queue. If the LeafQueue is interminably tied to CS, we may need to update the references in LeafQueue to use CapacityScheduler instead of CapacitySchedulerContext, this will make such dependency clear and avoid future confusions. I haven't made this interface change in the attached patch, as it requires more changes to other components, but if we agree about it, I can do it in a following issue.
        Ahmed Radwan made changes -
        Attachment MAPREDUCE-2788_rev3.patch [ 12499089 ]
        Ahmed Radwan made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12499089/MAPREDUCE-2788_rev3.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 javadoc. The javadoc tool appears to have generated 20 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1023//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1023//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12499089/MAPREDUCE-2788_rev3.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc. The javadoc tool appears to have generated 20 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1023//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1023//console This message is automatically generated.
        Hide
        Arun C Murthy added a comment -

        Ahmed - not sure when this happened, but looks like CS.allocate already normalizes the incoming request.

        Thus, we might only need to fix FifoScheduler.normalizeRequest?

        We should pull out normalizeRequest to SchedulerUtils and use it from both places?

        Show
        Arun C Murthy added a comment - Ahmed - not sure when this happened, but looks like CS.allocate already normalizes the incoming request. Thus, we might only need to fix FifoScheduler.normalizeRequest? We should pull out normalizeRequest to SchedulerUtils and use it from both places?
        Arun C Murthy made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Hide
        Arun C Murthy added a comment -

        Sorry, I missed the fact that CS seems to be fine now... a while since I looked at it. Apologies!

        Show
        Arun C Murthy added a comment - Sorry, I missed the fact that CS seems to be fine now... a while since I looked at it. Apologies!
        Hide
        Ahmed Radwan added a comment -

        Yes CS.allocate() already calls normalizeRequests(..), no need to change anything there. But the call scenario described in my previous comment is the one causing the problem, not CS.allocate(..), it is the CS.addApplication(..).

        The latest patch fix to CS.addApplication(..) to also normalize requests is the one required. What do you think?

        Show
        Ahmed Radwan added a comment - Yes CS.allocate() already calls normalizeRequests(..), no need to change anything there. But the call scenario described in my previous comment is the one causing the problem, not CS.allocate(..), it is the CS.addApplication(..). The latest patch fix to CS.addApplication(..) to also normalize requests is the one required. What do you think?
        Hide
        Arun C Murthy added a comment -

        Ahmed, when we add a new SchedulerApp it doesn't have any resource requests until the ApplicationsManager (or later, the ApplicationMaster itself) calls CS.allocate. Hence it isn't necessary to normalize requests. Makes sense?

        Show
        Arun C Murthy added a comment - Ahmed, when we add a new SchedulerApp it doesn't have any resource requests until the ApplicationsManager (or later, the ApplicationMaster itself) calls CS.allocate. Hence it isn't necessary to normalize requests. Makes sense?
        Hide
        Arun C Murthy added a comment -

        Changing the title to reflect the existing bug.

        Show
        Arun C Murthy added a comment - Changing the title to reflect the existing bug.
        Arun C Murthy made changes -
        Summary LeafQueue.assignContainer() can cause a crash if request.getCapability().getMemory() == 0 Normalize requests in FifoScheduler.allocate to prevent NPEs later
        Priority Critical [ 2 ] Major [ 3 ]
        Hide
        Ahmed Radwan added a comment -

        Yes, it makes sense now. Thanks

        Show
        Ahmed Radwan added a comment - Yes, it makes sense now. Thanks
        Hide
        Ahmed Radwan added a comment -

        The attached new patch pulls out requests' normalization methods to SchedulerUtils, and both FS and CS now use them. FS.allocate() already calls normalizeRequests(), but the previous implementation for normalizeRequest() in FS doesn't rectify the case where memory==0. This is now fixed. Please let me know if you have further comments.

        Show
        Ahmed Radwan added a comment - The attached new patch pulls out requests' normalization methods to SchedulerUtils, and both FS and CS now use them. FS.allocate() already calls normalizeRequests(), but the previous implementation for normalizeRequest() in FS doesn't rectify the case where memory==0. This is now fixed. Please let me know if you have further comments.
        Ahmed Radwan made changes -
        Attachment MAPREDUCE-2788_rev4.patch [ 12499138 ]
        Ahmed Radwan made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12499138/MAPREDUCE-2788_rev4.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1032//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1032//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12499138/MAPREDUCE-2788_rev4.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1032//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1032//console This message is automatically generated.
        Hide
        Mahadev konar added a comment -

        Ahmed,
        Could you please add a test case with this patch?

        Show
        Mahadev konar added a comment - Ahmed, Could you please add a test case with this patch?
        Mahadev konar made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Hide
        Ahmed Radwan added a comment -

        Will do, thanks Mahadev.

        Show
        Ahmed Radwan added a comment - Will do, thanks Mahadev.
        Hide
        Ahmed Radwan added a comment -

        Added a simple test case to verify the request normalization behavior for different valid and invalid memory values.

        Show
        Ahmed Radwan added a comment - Added a simple test case to verify the request normalization behavior for different valid and invalid memory values.
        Ahmed Radwan made changes -
        Attachment MAPREDUCE-2788_rev5.patch [ 12499662 ]
        Ahmed Radwan made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12499662/MAPREDUCE-2788_rev5.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 2 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 160 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1067//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1067//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1067//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1067//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1067//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12499662/MAPREDUCE-2788_rev5.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 160 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1067//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1067//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1067//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1067//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1067//console This message is automatically generated.
        Hide
        Arun C Murthy added a comment -

        I just committed this. Thanks Ahmed!

        Show
        Arun C Murthy added a comment - I just committed this. Thanks Ahmed!
        Arun C Murthy made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Arun C Murthy made changes -
        Fix Version/s 0.23.0 [ 12315570 ]
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Commit #25 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/25/)
        Merge -c 1186467 from trunk to branch-0.23 to complete fix for MAPREDUCE-2788.

        acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1186468
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerUtils.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Commit #25 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/25/ ) Merge -c 1186467 from trunk to branch-0.23 to complete fix for MAPREDUCE-2788 . acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1186468 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerUtils.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #1111 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1111/)
        MAPREDUCE-2788. Normalize resource requests in FifoScheduler appropriately. Contributed by Ahmed Radwan.

        acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1186467
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerUtils.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1111 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1111/ ) MAPREDUCE-2788 . Normalize resource requests in FifoScheduler appropriately. Contributed by Ahmed Radwan. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1186467 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerUtils.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #1190 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1190/)
        MAPREDUCE-2788. Normalize resource requests in FifoScheduler appropriately. Contributed by Ahmed Radwan.

        acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1186467
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerUtils.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1190 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1190/ ) MAPREDUCE-2788 . Normalize resource requests in FifoScheduler appropriately. Contributed by Ahmed Radwan. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1186467 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerUtils.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-0.23-Commit #25 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/25/)
        Merge -c 1186467 from trunk to branch-0.23 to complete fix for MAPREDUCE-2788.

        acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1186468
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerUtils.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-0.23-Commit #25 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/25/ ) Merge -c 1186467 from trunk to branch-0.23 to complete fix for MAPREDUCE-2788 . acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1186468 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerUtils.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-0.23-Commit #26 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/26/)
        Merge -c 1186467 from trunk to branch-0.23 to complete fix for MAPREDUCE-2788.

        acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1186468
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerUtils.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Commit #26 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/26/ ) Merge -c 1186467 from trunk to branch-0.23 to complete fix for MAPREDUCE-2788 . acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1186468 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerUtils.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #1130 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1130/)
        MAPREDUCE-2788. Normalize resource requests in FifoScheduler appropriately. Contributed by Ahmed Radwan.

        acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1186467
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerUtils.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1130 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1130/ ) MAPREDUCE-2788 . Normalize resource requests in FifoScheduler appropriately. Contributed by Ahmed Radwan. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1186467 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerUtils.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #45 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/45/)
        Merge -c 1186467 from trunk to branch-0.23 to complete fix for MAPREDUCE-2788.

        acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1186468
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerUtils.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #45 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/45/ ) Merge -c 1186467 from trunk to branch-0.23 to complete fix for MAPREDUCE-2788 . acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1186468 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerUtils.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-0.23-Build #57 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/57/)
        Merge -c 1186467 from trunk to branch-0.23 to complete fix for MAPREDUCE-2788.

        acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1186468
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerUtils.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #57 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/57/ ) Merge -c 1186467 from trunk to branch-0.23 to complete fix for MAPREDUCE-2788 . acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1186468 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerUtils.java
        Arun C Murthy made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Ahmed Radwan
            Reporter:
            Ahmed Radwan
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development