Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1018

Document changes to the memory management and scheduling model

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.21.0
    • Component/s: documentation
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Updated documentation related to (1) the changes in the configuration for memory management of tasks and (2) the modifications in scheduling model of CapacityTaskScheduler to include memory requirement by tasks.

      Description

      There were changes done for the configuration, monitoring and scheduling of high ram jobs. This must be documented in the mapred-defaults.xml and also on forrest documentation

      1. MAPRED-1018-1.patch
        13 kB
        rahul k singh
      2. MAPRED-1018-2.patch
        15 kB
        rahul k singh
      3. MAPRED-1018-3.patch
        20 kB
        rahul k singh
      4. MAPRED-1018-4.patch.txt
        22 kB
        rahul k singh
      5. MAPRED-1018-5.patch.txt
        22 kB
        Hemanth Yamijala
      6. MAPRED-1018-6.patch.txt
        22 kB
        Hemanth Yamijala
      7. MAPRED-1018-7.patch.txt
        31 kB
        Hemanth Yamijala
      8. MAPRED-1018-8.patch.txt
        34 kB
        Hemanth Yamijala
      9. MAPRED-1018-9.patch.txt
        17 kB
        Hemanth Yamijala
      10. MAPRED-1018-commons.patch
        8 kB
        rahul k singh

        Issue Links

          Activity

          Hide
          Vinod Kumar Vavilapalli added a comment -

          I just committed this branch 0.21 and trunk. Thanks Hemanth!

          Thanks also to Rahul Singh for earlier versions of the patch.

          Show
          Vinod Kumar Vavilapalli added a comment - I just committed this branch 0.21 and trunk. Thanks Hemanth! Thanks also to Rahul Singh for earlier versions of the patch.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          +1 for the patch from my side too.

          This is a documentation patch. Hudson is being insane. I ran docs and binary targets successfully and verified the content look of the web pages generated. I'm going to check this in.

          Show
          Vinod Kumar Vavilapalli added a comment - +1 for the patch from my side too. This is a documentation patch. Hudson is being insane. I ran docs and binary targets successfully and verified the content look of the web pages generated. I'm going to check this in.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12446832/MAPRED-1018-9.patch.txt
          against trunk revision 953976.

          +1 @author. The patch does not contain any @author tags.

          +0 tests included. The patch appears to be a documentation patch that doesn't require tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/567/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/567/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/567/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/567/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12446832/MAPRED-1018-9.patch.txt against trunk revision 953976. +1 @author. The patch does not contain any @author tags. +0 tests included. The patch appears to be a documentation patch that doesn't require tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/567/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/567/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/567/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/567/console This message is automatically generated.
          Hide
          Hemanth Yamijala added a comment -

          Running through Hudson again.

          Show
          Hemanth Yamijala added a comment - Running through Hudson again.
          Hide
          Tom White added a comment -

          +1

          Show
          Tom White added a comment - +1
          Hide
          Hemanth Yamijala added a comment -

          Attaching a new patch that merges with trunk, and incorporates Vinod's review comments. One of the links will work only when the corresponding changes to cluster_setup are committed to Common. This is expected.

          Show
          Hemanth Yamijala added a comment - Attaching a new patch that merges with trunk, and incorporates Vinod's review comments. One of the links will work only when the corresponding changes to cluster_setup are committed to Common. This is expected.
          Hide
          Hemanth Yamijala added a comment -

          Vinod, I was in the process of generating two patches; thanks for taking extra steps. That will fix the broken links etc. I will also work on your comment related to occupying multiple slots. In cluster_setup.xml, I had removed the documentation on java.opts and ulimits specifically. I have linked them saying they are described in detail in mapred_tutorial, where I have indeed described them in detail. Having the same definitions at two places would be very difficult to maintain. Once you have looked at the changes for mapred_tutorial, you could tell if this suffices or not.

          Please expect a new patch in a day or so.

          Show
          Hemanth Yamijala added a comment - Vinod, I was in the process of generating two patches; thanks for taking extra steps. That will fix the broken links etc. I will also work on your comment related to occupying multiple slots. In cluster_setup.xml, I had removed the documentation on java.opts and ulimits specifically. I have linked them saying they are described in detail in mapred_tutorial, where I have indeed described them in detail. Having the same definitions at two places would be very difficult to maintain. Once you have looked at the changes for mapred_tutorial, you could tell if this suffices or not. Please expect a new patch in a day or so.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          This patch needs to be split across projects after MAPREDUCE-1404. I tried doing it myself for the sake of reviewing.

          Few links are broken in capacity_scheduler.html and cluster_setup.html.

          I couldn't merge changes to mapred_tutorial.xml easily, and so couldn't review it at all.

          The changes in capacity_scheduler.xml and cluster_setup.xml mostly look good.

          capacity_scheduler.xml

          • "That is, the task is scheduled only if the following constraint is satisfied:"
            Can we also somehow mention here that high memory jobs can occupy multiple slots? I felt that isn't getting conveyed here, but was clear in Rahul's patch.
            If we do this, then we should also mention setup and cleanup tasks always occupying only a single slot.

          cluster_setup.xml

          • The documentation related to setting java.opts and ulimits w.r.t memory usage is (accidentally?) removed by your patch.
          Show
          Vinod Kumar Vavilapalli added a comment - This patch needs to be split across projects after MAPREDUCE-1404 . I tried doing it myself for the sake of reviewing. Few links are broken in capacity_scheduler.html and cluster_setup.html. I couldn't merge changes to mapred_tutorial.xml easily, and so couldn't review it at all. The changes in capacity_scheduler.xml and cluster_setup.xml mostly look good. capacity_scheduler.xml "That is, the task is scheduled only if the following constraint is satisfied:" Can we also somehow mention here that high memory jobs can occupy multiple slots? I felt that isn't getting conveyed here, but was clear in Rahul's patch. If we do this, then we should also mention setup and cleanup tasks always occupying only a single slot. cluster_setup.xml The documentation related to setting java.opts and ulimits w.r.t memory usage is (accidentally?) removed by your patch.
          Hide
          Hemanth Yamijala added a comment -

          Tom, I can certainly stage this to go in after MAPREDUCE-1404. Also, I've made changes in this patch itself to include the missing documentation for MAPREDUCE-1221. So, there's no need for a separate issue.

          Vinod, thanks for taking a look at it.

          Show
          Hemanth Yamijala added a comment - Tom, I can certainly stage this to go in after MAPREDUCE-1404 . Also, I've made changes in this patch itself to include the missing documentation for MAPREDUCE-1221 . So, there's no need for a separate issue. Vinod, thanks for taking a look at it.
          Hide
          Hemanth Yamijala added a comment -

          Attaching a patch that makes changes in documentation for physical memory monitoring and configuration as well. Over the last patch, the main changes are:

          • Describe the user and admin options for physical memory usage in the mapred-tutorial and cluster-setup guides respectively. I also included a small note on when physical memory monitoring was found useful based on discussions in MAPREDUCE-1221.
          • Qualified memory as virtual or physical or both at relevant places so we are clear what we are talking about.
          Show
          Hemanth Yamijala added a comment - Attaching a patch that makes changes in documentation for physical memory monitoring and configuration as well. Over the last patch, the main changes are: Describe the user and admin options for physical memory usage in the mapred-tutorial and cluster-setup guides respectively. I also included a small note on when physical memory monitoring was found useful based on discussions in MAPREDUCE-1221 . Qualified memory as virtual or physical or both at relevant places so we are clear what we are talking about.
          Hide
          Tom White added a comment -

          Hemanth, This is going to conflict with MAPREDUCE-1404, which moves cluster setup and commands manual pages to common. Would you be able to stage this one to go in after MAPREDUCE-1404?

          One thing I've not included is the documentation on the RSS based monitoring introduced in MAPREDUCE-1221. I am not yet familiar with that part of the code. Also, this patch itself looks reasonably big. Hence, I would request for those changes to be incorporated as a follow-up, though they should be treated as a blocker for the 0.21 release as well. Thoughts ?

          Fine to have as a separate issue, but I don't think it would stop a release going out (we could add it in a point release if it didn't make it into the .0 release).

          Show
          Tom White added a comment - Hemanth, This is going to conflict with MAPREDUCE-1404 , which moves cluster setup and commands manual pages to common. Would you be able to stage this one to go in after MAPREDUCE-1404 ? One thing I've not included is the documentation on the RSS based monitoring introduced in MAPREDUCE-1221 . I am not yet familiar with that part of the code. Also, this patch itself looks reasonably big. Hence, I would request for those changes to be incorporated as a follow-up, though they should be treated as a blocker for the 0.21 release as well. Thoughts ? Fine to have as a separate issue, but I don't think it would stop a release going out (we could add it in a point release if it didn't make it into the .0 release).
          Hide
          Vinod Kumar Vavilapalli added a comment -

          I suspect it will be ready in a day. If no one has started looking at this, I will upload a new patch mostly by tomorrow.

          Please do so. I am now going through my backlog, been quite a while. I'll start looking at your earlier patch just in case to see if you've missed anything.

          Show
          Vinod Kumar Vavilapalli added a comment - I suspect it will be ready in a day. If no one has started looking at this, I will upload a new patch mostly by tomorrow. Please do so. I am now going through my backlog, been quite a while. I'll start looking at your earlier patch just in case to see if you've missed anything.
          Hide
          Hemanth Yamijala added a comment -

          One thing I've not included is the documentation on the RSS based monitoring introduced in MAPREDUCE-1221.

          I am working on a patch to include the RSS based monitoring changes into the documentation as well. I suspect it will be ready in a day. If no one has started looking at this, I will upload a new patch mostly by tomorrow. Please let me know if you prefer otherwise.

          Show
          Hemanth Yamijala added a comment - One thing I've not included is the documentation on the RSS based monitoring introduced in MAPREDUCE-1221 . I am working on a patch to include the RSS based monitoring changes into the documentation as well. I suspect it will be ready in a day. If no one has started looking at this, I will upload a new patch mostly by tomorrow. Please let me know if you prefer otherwise.
          Hide
          Hemanth Yamijala added a comment -

          The attached patch [MAPRED-1018-7.patch.txt], makes changes to the memory monitoring, configuration and scheduling sections. The changes over earlier patches are primarily in the nature of presentation.

          A brief summary:

          • Mapreduce Tutorial: This now describes all the Job specific memory configuration options. The expectation is that users who have questions about how to configure memory requirements for their jobs can get answers here.
          • Cluster Setup: This describes memory monitoring, links to the job specific options in the Mapreduce tutorial, and describes in detail how to configure cluster specific memory configuration options. I've removed the duplication of describing the job specific options in both the places, since I felt it will be a problem to maintain changes. This describes memory related aspects from an administrator point of view.
          • Capacity Scheduler: This describes memory based scheduling. Instead of spelling the precise algorithm out, I have given a gist of how the scheduler works. The description is more in terms of what the scheduler does, rather than how it does it.

          Request a review of the same. Please ensure that all the required content is captured, and I've not missed out anything when reorganizing. Also, please check if the documentation is clear to understand.

          One thing I've not included is the documentation on the RSS based monitoring introduced in MAPREDUCE-1221. I am not yet familiar with that part of the code. Also, this patch itself looks reasonably big. Hence, I would request for those changes to be incorporated as a follow-up, though they should be treated as a blocker for the 0.21 release as well. Thoughts ?

          Show
          Hemanth Yamijala added a comment - The attached patch [MAPRED-1018-7.patch.txt] , makes changes to the memory monitoring, configuration and scheduling sections. The changes over earlier patches are primarily in the nature of presentation. A brief summary: Mapreduce Tutorial: This now describes all the Job specific memory configuration options. The expectation is that users who have questions about how to configure memory requirements for their jobs can get answers here. Cluster Setup: This describes memory monitoring, links to the job specific options in the Mapreduce tutorial, and describes in detail how to configure cluster specific memory configuration options. I've removed the duplication of describing the job specific options in both the places, since I felt it will be a problem to maintain changes. This describes memory related aspects from an administrator point of view. Capacity Scheduler: This describes memory based scheduling. Instead of spelling the precise algorithm out, I have given a gist of how the scheduler works. The description is more in terms of what the scheduler does, rather than how it does it. Request a review of the same. Please ensure that all the required content is captured, and I've not missed out anything when reorganizing. Also, please check if the documentation is clear to understand. One thing I've not included is the documentation on the RSS based monitoring introduced in MAPREDUCE-1221 . I am not yet familiar with that part of the code. Also, this patch itself looks reasonably big. Hence, I would request for those changes to be incorporated as a follow-up, though they should be treated as a blocker for the 0.21 release as well. Thoughts ?
          Hide
          Hemanth Yamijala added a comment -

          Assigning to myself to take forward. I've started work on it, but please bear with slow progress.

          Show
          Hemanth Yamijala added a comment - Assigning to myself to take forward. I've started work on it, but please bear with slow progress.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          I will reassign this to myself to take it forward, if no one has objections.

          Please do so Hemanth. I can, may be, help with patch review, given I did the first review. Thanks!

          Show
          Vinod Kumar Vavilapalli added a comment - I will reassign this to myself to take it forward, if no one has objections. Please do so Hemanth. I can, may be, help with patch review, given I did the first review. Thanks!
          Hide
          Hemanth Yamijala added a comment -

          The previous patch failed to generate documentation due to Forrest compile errors. I am attaching a new patch that fixes these. However, glancing over the changes, I feel we can make some changes. I will reassign this to myself to take it forward, if no one has objections.

          Show
          Hemanth Yamijala added a comment - The previous patch failed to generate documentation due to Forrest compile errors. I am attaching a new patch that fixes these. However, glancing over the changes, I feel we can make some changes. I will reassign this to myself to take it forward, if no one has objections.
          Hide
          Hemanth Yamijala added a comment -

          I am taking a swipe at finishing this blocker. Attaching a new patch that applies to trunk, since the old one did not.

          Show
          Hemanth Yamijala added a comment - I am taking a swipe at finishing this blocker. Attaching a new patch that applies to trunk, since the old one did not.
          Hide
          rahul k singh added a comment -

          implemented the suggestions above

          Show
          rahul k singh added a comment - implemented the suggestions above
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Looked at the latest patch. I've some more comments. Some of the following may not have been introduced by your patch though.

          cluster_setup.html

          • Statement I: Users can, optionally, specify the MEM task-limit per job. If no such limit is provided, a default limit is used. A node-limit can be set per node.
            We don't have default limits anymore. So the statement "If no such limit is provided, a default limit is used." can be removed.
          • Node-limit cannot be set directly anymore. So, we should define the node limit here by saying "Node-limit of total memory usage for tasks is given by Node-limit = mapreduce.tasktracker.map.tasks.maximum * mapreduce.cluster.mapmemory.mb + mapreduce.tasktracker.reduce.tasks.maximum * mapreduce.cluster.mapmemory.mb".
          • To enable monitoring for a TT, the following parameters all need to be set:
            This is not true w.r.t job configuration. So this table should only have TT configuration. And move job configuration to another table. We should move the above statement to later and thus the whole section would look like

            To enable monitoring for a TT, the following parameters all need to be set:
            TABLE:I with TT parameters
            Node-limit of total memory usage for tasks is given by Node-limit = mapreduce.tasktracker.map.tasks.maximum * mapreduce.cluster.mapmemory.mb + mapreduce.tasktracker.reduce.tasks.maximum * mapreduce.cluster.mapmemory.mb"
            Users can, optionally, specify the MEM task-limit per job.
            TABLE II with Job parameters

          • 2. Periodically, the TT checks the following:
            Should be "If memory monitoring is enabled, the TT does the following periodically:"

          mapred_tutorial.html:

          • Memory management section defines the job parameters again here. But "mapreduce.map.memory.mb" is repeated twice, one of them should be "mapreduce.reduce.memory.mb"
          • Users can choose to override default limits of memory enforced by the task tracker, if memory management is enabled. Users can set the following parameter per job:
            Please modify "memory management" to be "memory monitoring" and link it to the monitoring section in cluster_setup.html

          capacity_scheduler.html:

          • Please rename the section name to "Memory-based task-scheduling"
          • The Capacity Scheduler supports scheduling of tasks on a TaskTracker(TT) based on a job's memory requirements and the availability of RAM and Virtual Memory (VMEM) on the TT node.
            As my previous review comments mentioned, support for RAM availability is no longer there. So it should read "....and the availability of Virtual Memory (VMEM) on the TT node. There isn't any need for distinguishing VMEM from physical memory w.r.t tasks. Depending on a site's requirements, the configuration can be set depending on whether one wants tasks to go beyond physical memory or not."
          • "See the MapReduce Tutorial for details on how the TT monitors memory usage."
            This should actually point to cluster_setup.html. Previous review comment missed.
          • "Currently the memory based scheduling is only supported in Linux platform."
            This isn't quite right. It should be "Memory based scheduling primarily exists to avoid memory pressure by tasks on a TT and thus is dependent on TT-memory monitoring which currently is only supported in Linux platform."
          • "1. The absence of any one or more of four config parameters or -1 being set as value of any of the parameters, mapreduce.cluster.mapmemory.mb, mapreduce.cluster.reducememory.mb, or mapreduce.jobtracker.maxmapmemory.mb, mapreduce.jobtracker.maxreducememory.mb disables memory-based scheduling, just as it disables memory monitoring for a TT. These config parameters are described in the MapReduce Tutorial. "
            This can be greatly simplified to "The configuration properties mapreduce.cluster.mapmemory.mb, mapreduce.cluster.reducememory.mb, or mapreduce.jobtracker.maxmapmemory.mb, mapreduce.jobtracker.maxreducememory.mb are used to enable/disable memory based scheduling. The absence of being set as -1 of any one of these properties disables memory-based scheduling, just as it disables monitoring for a TT. These parameters are described in the Cluster-setup <a href="cluster_setup.html#memory_monitoring">memory-monitoring section</a>."
          • The second statement that describes scheduling can be greatly simplified by writing it as a list of points, like my previous review described. Also, we haven't introduces reservations anywhere else, so that part also needs to be explained so and clear. Roughly,
            • Point 2 in Working of scheduling
              Total = numSlots * PerSlotMemSize.
              Used = Sigma(numSlotsPerTask * PerSlotMemSize)
              if (can fit), schedule. Otherwise reserve. Reserve why? what? How many?

            That was just a rough cut, but should give you an idea

          Show
          Vinod Kumar Vavilapalli added a comment - Looked at the latest patch. I've some more comments. Some of the following may not have been introduced by your patch though. cluster_setup.html Statement I: Users can, optionally, specify the MEM task-limit per job. If no such limit is provided, a default limit is used. A node-limit can be set per node. We don't have default limits anymore. So the statement "If no such limit is provided, a default limit is used." can be removed. Node-limit cannot be set directly anymore. So, we should define the node limit here by saying "Node-limit of total memory usage for tasks is given by Node-limit = mapreduce.tasktracker.map.tasks.maximum * mapreduce.cluster.mapmemory.mb + mapreduce.tasktracker.reduce.tasks.maximum * mapreduce.cluster.mapmemory.mb" . To enable monitoring for a TT, the following parameters all need to be set: This is not true w.r.t job configuration. So this table should only have TT configuration. And move job configuration to another table. We should move the above statement to later and thus the whole section would look like To enable monitoring for a TT, the following parameters all need to be set: TABLE:I with TT parameters Node-limit of total memory usage for tasks is given by Node-limit = mapreduce.tasktracker.map.tasks.maximum * mapreduce.cluster.mapmemory.mb + mapreduce.tasktracker.reduce.tasks.maximum * mapreduce.cluster.mapmemory.mb" Users can, optionally, specify the MEM task-limit per job. TABLE II with Job parameters 2. Periodically, the TT checks the following: Should be "If memory monitoring is enabled, the TT does the following periodically:" mapred_tutorial.html: Memory management section defines the job parameters again here. But "mapreduce.map.memory.mb" is repeated twice, one of them should be "mapreduce.reduce.memory.mb" Users can choose to override default limits of memory enforced by the task tracker, if memory management is enabled. Users can set the following parameter per job: Please modify "memory management" to be "memory monitoring" and link it to the monitoring section in cluster_setup.html capacity_scheduler.html: Please rename the section name to "Memory-based task-scheduling" The Capacity Scheduler supports scheduling of tasks on a TaskTracker(TT) based on a job's memory requirements and the availability of RAM and Virtual Memory (VMEM) on the TT node. As my previous review comments mentioned, support for RAM availability is no longer there. So it should read "....and the availability of Virtual Memory (VMEM) on the TT node. There isn't any need for distinguishing VMEM from physical memory w.r.t tasks. Depending on a site's requirements, the configuration can be set depending on whether one wants tasks to go beyond physical memory or not." "See the MapReduce Tutorial for details on how the TT monitors memory usage." This should actually point to cluster_setup.html. Previous review comment missed. "Currently the memory based scheduling is only supported in Linux platform." This isn't quite right. It should be "Memory based scheduling primarily exists to avoid memory pressure by tasks on a TT and thus is dependent on TT-memory monitoring which currently is only supported in Linux platform." "1. The absence of any one or more of four config parameters or -1 being set as value of any of the parameters, mapreduce.cluster.mapmemory.mb, mapreduce.cluster.reducememory.mb, or mapreduce.jobtracker.maxmapmemory.mb, mapreduce.jobtracker.maxreducememory.mb disables memory-based scheduling, just as it disables memory monitoring for a TT. These config parameters are described in the MapReduce Tutorial. " This can be greatly simplified to "The configuration properties mapreduce.cluster.mapmemory.mb, mapreduce.cluster.reducememory.mb, or mapreduce.jobtracker.maxmapmemory.mb, mapreduce.jobtracker.maxreducememory.mb are used to enable/disable memory based scheduling. The absence of being set as -1 of any one of these properties disables memory-based scheduling, just as it disables monitoring for a TT. These parameters are described in the Cluster-setup <a href="cluster_setup.html#memory_monitoring">memory-monitoring section</a>." The second statement that describes scheduling can be greatly simplified by writing it as a list of points, like my previous review described. Also, we haven't introduces reservations anywhere else, so that part also needs to be explained so and clear. Roughly, Point 2 in Working of scheduling Total = numSlots * PerSlotMemSize. Used = Sigma(numSlotsPerTask * PerSlotMemSize) if (can fit), schedule. Otherwise reserve. Reserve why? what? How many? That was just a rough cut, but should give you an idea
          Hide
          rahul k singh added a comment -

          incorporated vinod's comments.

          Show
          rahul k singh added a comment - incorporated vinod's comments.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Already so many other mapreduce issues have only modified cluster-setup.xml, the one in mapreduce project. Rahul mentioned offline that forrest documentation is not getting generated in mapreduce sub-project. Assuming we'll address that in a separate issue, I propose we have only one patch - the mapred one.

          • The mapred Patch has git prefixes which need to be removed
          • Monitoring/Scheduling based on RAM is completely removed. So remove the references too. Just add a note saying that (quoting from HADOOP-5881) there isn't any need for distinguishing vmem from physical memory w.r.t configuration. Depending on a site's requirements, the configuration items can reflect whether one wants tasks to go beyond physical memory or not.

          cluster_setup.html

          • All config names should be renamed to the new names. Of-course this means a slightly different patch for 0.20 - which we will come to after the patch
            for trunk's done
          • mapred. {map|reduce}

            .child.ulimit also need to be renamed

          • What happens when monitoring is enabled, but job has -1?
          • Memory-monitoring is no longer defined in terms of per-task-limit and per-node-limit. It is now driven by per-slot-size and number of slots. We should use these new terms through-out.
          • "Before getting into details, consider the following additional memory-related parameters than can be configured to enable better scheduling:"\
            The above line is no longer needed.

          capacity_scheduler.html

          • Feature for monitoring RAM no more. Remove all references.
          • Working of scheduling
            • Point 1: 4 parameters, not three. Parameters described in cluster_setup. vmem.reserved no more used.
            • Point 2: This is changed completely. No more offsets.
              Total = numSlots * PerSlotMemSize.
              Used = Sigma(numSlotsPerTask * PerSlotMemSize)
            • Point 3: JT now rejects the jobs, not the scheduler.
          • "See the MapReduce Tutorial for details on how the TT monitors memory usage."
            "See cluster_setup" instead?
          • Need to update mapred_tutorial.html's memory management section. Aslo need a reference to this in both cluster_setup.html as well as capacity_scheduler.html
          • Another point I've already mentioned on the JIRA.
            "Along with everything else, we should document that job setup and job cleanup tasks of all jobs, either requiring or not requiring high memory for their maps and reduces, still run on a single slot.
          Show
          Vinod Kumar Vavilapalli added a comment - Already so many other mapreduce issues have only modified cluster-setup.xml, the one in mapreduce project. Rahul mentioned offline that forrest documentation is not getting generated in mapreduce sub-project. Assuming we'll address that in a separate issue, I propose we have only one patch - the mapred one. The mapred Patch has git prefixes which need to be removed Monitoring/Scheduling based on RAM is completely removed. So remove the references too. Just add a note saying that (quoting from HADOOP-5881 ) there isn't any need for distinguishing vmem from physical memory w.r.t configuration. Depending on a site's requirements, the configuration items can reflect whether one wants tasks to go beyond physical memory or not. cluster_setup.html All config names should be renamed to the new names. Of-course this means a slightly different patch for 0.20 - which we will come to after the patch for trunk's done mapred. {map|reduce} .child.ulimit also need to be renamed What happens when monitoring is enabled, but job has -1? Memory-monitoring is no longer defined in terms of per-task-limit and per-node-limit. It is now driven by per-slot-size and number of slots. We should use these new terms through-out. "Before getting into details, consider the following additional memory-related parameters than can be configured to enable better scheduling:"\ The above line is no longer needed. capacity_scheduler.html Feature for monitoring RAM no more. Remove all references. Working of scheduling Point 1: 4 parameters, not three. Parameters described in cluster_setup. vmem.reserved no more used. Point 2: This is changed completely. No more offsets. Total = numSlots * PerSlotMemSize. Used = Sigma(numSlotsPerTask * PerSlotMemSize) Point 3: JT now rejects the jobs, not the scheduler. "See the MapReduce Tutorial for details on how the TT monitors memory usage." "See cluster_setup" instead? Need to update mapred_tutorial.html's memory management section. Aslo need a reference to this in both cluster_setup.html as well as capacity_scheduler.html Another point I've already mentioned on the JIRA. "Along with everything else, we should document that job setup and job cleanup tasks of all jobs, either requiring or not requiring high memory for their maps and reduces, still run on a single slot.
          Hide
          rahul k singh added a comment -

          patch for commons , need to open a seperate jira to handle this patch.
          This patch is for cluster_set.xml

          Show
          rahul k singh added a comment - patch for commons , need to open a seperate jira to handle this patch. This patch is for cluster_set.xml
          Hide
          rahul k singh added a comment -

          first cut

          Show
          rahul k singh added a comment - first cut
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Along with everything else, we should document that job setup and job cleanup tasks of all jobs, either requiring or not requiring high memory for their maps and reduces, still run on a single slot.

          Show
          Vinod Kumar Vavilapalli added a comment - Along with everything else, we should document that job setup and job cleanup tasks of all jobs, either requiring or not requiring high memory for their maps and reduces, still run on a single slot.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          I think this should be a blocker for 0.21.

          Show
          Vinod Kumar Vavilapalli added a comment - I think this should be a blocker for 0.21.

            People

            • Assignee:
              Hemanth Yamijala
              Reporter:
              Hemanth Yamijala
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development