Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-550

Add support for scheduling jobs based on memory requirements to the Fairscheduler


    • Type: New Feature
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:


      In HADOOP-3759, we added the ability for users to specify jobs requesting for a certain amount of virtual memory. For e.g. users can say that their jobs require 2GB of memory to run. In HADOOP-4035, functionality was added to the capacity scheduler to schedule jobs based on this specified amount. This JIRA is to add similar support to the Fairshare scheduler.

      The basic use case is that there are jobs that require a certain known amount of virtual memory, usually more than the JVM's heap size. This happens specifically for streaming jobs that can launch several processes from the child. Without being aware of these requirements, if tasks are scheduled on nodes just based on available slots, they have a potential of affecting the other processes running on the node, or if memory protection features are enabled (HADOOP-3581), they could result in the task being killed by the tasktracker.

      The scheduler must take into account the requested amount of memory by the job, the amount of memory that can be committed to by a tracker, and schedule based on these inputs.




            • Assignee:
              yhemanth Hemanth Yamijala
            • Votes:
              0 Vote for this issue
              1 Start watching this issue


              • Created: