Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1221

Kill tasks on a node if the free physical memory on that machine falls below a configured threshold

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.21.0
    • Component/s: tasktracker
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The TaskTracker currently supports killing tasks if the virtual memory of a task exceeds a set of configured thresholds. I would like to extend this feature to enable killing tasks if the physical memory used by that task exceeds a certain threshold.

      On a certain operating system (guess?), if user space processes start using lots of memory, the machine hangs and dies quickly. This means that we would like to prevent map-reduce jobs from triggering this condition. From my understanding, the killing-based-on-virtual-memory-limits (HADOOP-5883) were designed to address this problem. This works well when most map-reduce jobs are Java jobs and have well-defined -Xmx parameters that specify the max virtual memory for each task. On the other hand, if each task forks off mappers/reducers written in other languages (python/php, etc), the total virtual memory usage of the process-subtree varies greatly. In these cases, it is better to use kill-tasks-using-physical-memory-limits.

      1. MAPREDUCE-1221-v6.txt
        28 kB
        Scott Chen
      2. MAPREDUCE-1221-v5.1.txt
        27 kB
        Scott Chen
      3. MAPREDUCE-1221-v4.patch
        26 kB
        Scott Chen
      4. MAPREDUCE-1221-v3.patch
        16 kB
        Scott Chen
      5. MAPREDUCE-1221-v2.patch
        16 kB
        Scott Chen
      6. MAPREDUCE-1221-v1.patch
        15 kB
        Scott Chen
      7. ASF.LICENSE.NOT.GRANTED--MAPREDUCE-1221-v5.txt
        27 kB
        Scott Chen

        Activity

          People

          • Assignee:
            Scott Chen
            Reporter:
            dhruba borthakur
          • Votes:
            0 Vote for this issue
            Watchers:
            22 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development