Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-550

Add support for scheduling jobs based on memory requirements to the Fairscheduler

    Details

    • Type: New Feature New Feature
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      In HADOOP-3759, we added the ability for users to specify jobs requesting for a certain amount of virtual memory. For e.g. users can say that their jobs require 2GB of memory to run. In HADOOP-4035, functionality was added to the capacity scheduler to schedule jobs based on this specified amount. This JIRA is to add similar support to the Fairshare scheduler.

      The basic use case is that there are jobs that require a certain known amount of virtual memory, usually more than the JVM's heap size. This happens specifically for streaming jobs that can launch several processes from the child. Without being aware of these requirements, if tasks are scheduled on nodes just based on available slots, they have a potential of affecting the other processes running on the node, or if memory protection features are enabled (HADOOP-3581), they could result in the task being killed by the tasktracker.

      The scheduler must take into account the requested amount of memory by the job, the amount of memory that can be committed to by a tracker, and schedule based on these inputs.

        Activity

        There are no comments yet on this issue.

          People

          • Assignee:
            Unassigned
            Reporter:
            Hemanth Yamijala
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:

              Development