Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2108

Allow TaskScheduler manage number slots on TaskTrackers

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

      Currently the map slots and reduce slots are managed by TaskTracker configuration.
      To change the task tracker slots, we need to restart the TaskTrackers.
      Also, for a non-uniform cluster, we have to deploy different sets of configuration.

      Now JobTracker holds the CPU and memory status of TaskTrackers (MAPREDUCE-1218).
      So it makes sense to just let JobTracker.taskScheduler decided the number of slots on each node.
      This way we can
      1. Change the number of slots dynamically without restarting TaskTracker
      2. Use different number of slots based on the resource of a TaskTracker

      To achieve this, we need to change the logic that we use totalMapSlots and totalReduceSlots in JobTracker.
      I think they are used in WebUI and speculativeCap.

      We will need to make JobTracker calculate these numbers from TaskScheduler and TaskTrackerStatus.
      TaskScheduler and TaskTracker can both hold their maximum slots. We pick the smaller one.

      Thoughts?

      Attachments

        1. MAPREDUCE-2108-v2.txt
          6 kB
          Scott Chen
        2. MAPREDUCE-2108.txt
          5 kB
          Scott Chen

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            schen Scott Chen
            schen Scott Chen
            Votes:
            0 Vote for this issue
            Watchers:
            13 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment