Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1221

Kill tasks on a node if the free physical memory on that machine falls below a configured threshold

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.22.0
    • 0.21.0
    • tasktracker
    • None
    • Reviewed

    Description

      The TaskTracker currently supports killing tasks if the virtual memory of a task exceeds a set of configured thresholds. I would like to extend this feature to enable killing tasks if the physical memory used by that task exceeds a certain threshold.

      On a certain operating system (guess?), if user space processes start using lots of memory, the machine hangs and dies quickly. This means that we would like to prevent map-reduce jobs from triggering this condition. From my understanding, the killing-based-on-virtual-memory-limits (HADOOP-5883) were designed to address this problem. This works well when most map-reduce jobs are Java jobs and have well-defined -Xmx parameters that specify the max virtual memory for each task. On the other hand, if each task forks off mappers/reducers written in other languages (python/php, etc), the total virtual memory usage of the process-subtree varies greatly. In these cases, it is better to use kill-tasks-using-physical-memory-limits.

      Attachments

        1. ASF.LICENSE.NOT.GRANTED--MAPREDUCE-1221-v5.txt
          27 kB
          Scott Chen
        2. MAPREDUCE-1221-v1.patch
          15 kB
          Scott Chen
        3. MAPREDUCE-1221-v2.patch
          16 kB
          Scott Chen
        4. MAPREDUCE-1221-v3.patch
          16 kB
          Scott Chen
        5. MAPREDUCE-1221-v4.patch
          26 kB
          Scott Chen
        6. MAPREDUCE-1221-v5.1.txt
          27 kB
          Scott Chen
        7. MAPREDUCE-1221-v6.txt
          28 kB
          Scott Chen

        Activity

          People

            schen Scott Chen
            dhruba Dhruba Borthakur
            Votes:
            0 Vote for this issue
            Watchers:
            22 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: