Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1181

Enforce RSS memory limit in TaskMemoryManagerThread

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Invalid
    • 0.20.1
    • 0.20.1
    • tasktracker
    • None

    Description

      TaskMemoryManagerThread will periodically check the rss memory usage of every task. If the memory usage exceeds the specified threshold, the task will be killed. Also if the total rss memory of all tasks exceeds (total amount of memory - specified reserved memory). The task with least progress will be killed to recover the reserved rss memory.

      This is similar to the virtual memory limit provided by TaskMemoryManagerThread. But now the limit is for rss memory. This new feature allow us to avoid page swapping which is prone to error.

      The following are the related configurations
      mapreduce.reduce.memory.rss.mb // RSS memory allowed for a reduce task
      mapreduce.map.memory.rss.mb // RSS memory allowed for a map task
      mapreduce.tasktracker.reserved.memory.rss.mb // RSS memory reserved (not for tasks) on a tasktracker

      Attachments

        1. MAPREDUCE-1181.patch
          39 kB
          Scott Chen

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            schen Scott Chen
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment