Details

    • Type: Sub-task Sub-task
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.23.0
    • Fix Version/s: None
    • Component/s: mrv2
    • Labels:
      None

      Description

      MRV2 currently does not take reduce disk usage into account when trying to schedule a container. For feature parity with the original map reduce it should be extended to allow for disk space requests within containers along with RAM requests. We then also need to port MAPREDUCE-2324 to the scheduler to allow it to avoid starvation of containers that might never get the resources that they need.

        Activity

        Hide
        Robert Joseph Evans added a comment -

        I have been looking at MR-279 and I want to do something similar to MR-2324 it is just not really set up to do it easily. The scheduling is split up between the resource manager and the application master. And in fact the resource manager/application master are completely ignoring disk utilization at this point.

        The plan is to add in disk utilization to the resources that the RM uses, and then have AM request both disk and RAM space for reduces with disk space based off of the size estimate currently used. Then inside the scheduler, which is the right place in my opinion to decide if a request is being starved or not, it would do just what MR-2324 does now but more generalized for all resource constraints, not just disk. This means that all schedulers would have to be modified to support this, but I can make the code generic so it should be fairly simple to do. I just need to dig into the MR-279 code to decide exactly how I want to insert this in. I should hopefully have a patch by mid next week.

        Show
        Robert Joseph Evans added a comment - I have been looking at MR-279 and I want to do something similar to MR-2324 it is just not really set up to do it easily. The scheduling is split up between the resource manager and the application master. And in fact the resource manager/application master are completely ignoring disk utilization at this point. The plan is to add in disk utilization to the resources that the RM uses, and then have AM request both disk and RAM space for reduces with disk space based off of the size estimate currently used. Then inside the scheduler, which is the right place in my opinion to decide if a request is being starved or not, it would do just what MR-2324 does now but more generalized for all resource constraints, not just disk. This means that all schedulers would have to be modified to support this, but I can make the code generic so it should be fairly simple to do. I just need to dig into the MR-279 code to decide exactly how I want to insert this in. I should hopefully have a patch by mid next week.
        Hide
        Arun C Murthy added a comment -

        I think this is the right direction...

        The AM should request for disk in addition to memory via ResourceRequest.

        In terms of timelines, I think it's a little early to right now. I'd rather wait for a .1 release to get this feature in to ensure we spend enough time stabilizing the core of the RM right now.

        Does that make sense? Thanks.

        Show
        Arun C Murthy added a comment - I think this is the right direction... The AM should request for disk in addition to memory via ResourceRequest. In terms of timelines, I think it's a little early to right now . I'd rather wait for a .1 release to get this feature in to ensure we spend enough time stabilizing the core of the RM right now. Does that make sense? Thanks.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #757 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/757/)
        MAPREDUCE-2723. Remove directly accessing FSNamesystem.LOG from TestCopyFiles and TestDistCh.

        szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1151349
        Files :

        • /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/tools/TestCopyFiles.java
        • /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/tools/TestDistCh.java
        • /hadoop/common/trunk/mapreduce/CHANGES.txt
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #757 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/757/ ) MAPREDUCE-2723 . Remove directly accessing FSNamesystem.LOG from TestCopyFiles and TestDistCh. szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1151349 Files : /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/tools/TestCopyFiles.java /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/tools/TestDistCh.java /hadoop/common/trunk/mapreduce/CHANGES.txt
        Hide
        Robert Joseph Evans added a comment -

        It looks like someone got the CHANGES.txt and checkin comments wrong. Those changes do not look like they have anything to do with this JIRA.

        Show
        Robert Joseph Evans added a comment - It looks like someone got the CHANGES.txt and checkin comments wrong. Those changes do not look like they have anything to do with this JIRA.
        Hide
        Robert Joseph Evans added a comment -

        Those notes are for MAPREDUCE-2732 (I just found it)

        Show
        Robert Joseph Evans added a comment - Those notes are for MAPREDUCE-2732 (I just found it)
        Hide
        Tsz Wo Nicholas Sze added a comment -

        > It looks like someone got the CHANGES.txt and checkin comments wrong. ...

        Oops, it was my mistake. Will fix CHANGE.txt. Thanks Robert for pointing it out.

        Show
        Tsz Wo Nicholas Sze added a comment - > It looks like someone got the CHANGES.txt and checkin comments wrong. ... Oops, it was my mistake. Will fix CHANGE.txt. Thanks Robert for pointing it out.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #759 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/759/)
        Fix a typo in CHANGE.txt: MAPREDUCE-2723 should be MAPREDUCE-2732.

        szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1151464
        Files :

        • /hadoop/common/trunk/mapreduce/CHANGES.txt
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #759 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/759/ ) Fix a typo in CHANGE.txt: MAPREDUCE-2723 should be MAPREDUCE-2732 . szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1151464 Files : /hadoop/common/trunk/mapreduce/CHANGES.txt
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #750 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/750/)
        MAPREDUCE-2723. Remove directly accessing FSNamesystem.LOG from TestCopyFiles and TestDistCh.

        szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1151349
        Files :

        • /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/tools/TestCopyFiles.java
        • /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/tools/TestDistCh.java
        • /hadoop/common/trunk/mapreduce/CHANGES.txt
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #750 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/750/ ) MAPREDUCE-2723 . Remove directly accessing FSNamesystem.LOG from TestCopyFiles and TestDistCh. szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1151349 Files : /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/tools/TestCopyFiles.java /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/tools/TestDistCh.java /hadoop/common/trunk/mapreduce/CHANGES.txt
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #751 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/751/)
        Fix a typo in CHANGE.txt: MAPREDUCE-2723 should be MAPREDUCE-2732.

        szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1151464
        Files :

        • /hadoop/common/trunk/mapreduce/CHANGES.txt
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #751 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/751/ ) Fix a typo in CHANGE.txt: MAPREDUCE-2723 should be MAPREDUCE-2732 . szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1151464 Files : /hadoop/common/trunk/mapreduce/CHANGES.txt

          People

          • Assignee:
            Robert Joseph Evans
            Reporter:
            Robert Joseph Evans
          • Votes:
            0 Vote for this issue
            Watchers:
            11 Start watching this issue

            Dates

            • Created:
              Updated:

              Development