Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1603

Add a plugin class for the TaskTracker to determine available slots

    Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Won't Fix
    • Affects Version/s: 0.22.0
    • Fix Version/s: None
    • Component/s: tasktracker
    • Labels:
      None

      Description

      Currently the #of available map and reduce slots is determined by the configuration. MAPREDUCE-922 has proposed working things out automatically, but that is going to depend a lot on the specific tasks -hard to get right for everyone.

      There is a Hadoop cluster near me that would like to use CPU time from other machines in the room, machines which cannot offer storage, but which will have spare CPU time when they aren't running code scheduled with a grid scheduler. The nodes could run a TT which would report a dynamic number of slots, the number depending upon the current grid workload.

      I propose we add a plugin point here, so that different people can develop plugin classes that determine the amount of available slots based on workload, RAM, CPU, power budget, thermal parameters, etc. Lots of space for customisation and improvement. And by having it as a plugin: people get to integrate with whatever datacentre schedulers they have without Hadoop itself needing to be altered: the base implementation would be as today: subtract the number of active map and reduce slots from the configured values, push that out.

        Activity

        Hide
        Allen Wittenauer added a comment -

        I like this idea a lot. It starts to crack open the door to much more advanced scheduling. For example, it'd be great to be able to pass up to the scheduling system if a machine is 32-bit or 64-bit, Windows or some Unix flavor, etc. This means at some other future point, jobs could request a particular environment.

        Show
        Allen Wittenauer added a comment - I like this idea a lot. It starts to crack open the door to much more advanced scheduling. For example, it'd be great to be able to pass up to the scheduling system if a machine is 32-bit or 64-bit, Windows or some Unix flavor, etc. This means at some other future point, jobs could request a particular environment.
        Hide
        Steve Loughran added a comment -

        -that would imply passing up machine metadata: cpu family/version, OS, etc. No reason why that couldn't be done, though you'd have to decide whether that is something you'd republish every heartbeat or just when the TT first registers. Of course, without the JT making decisions on where to route stuff based on those features, it's wasted effort. Which would imply you also need some plugin support for making the decisions as to where to run Mappers and Reducers; right now it's fairly straightforward: do it close to the data.

        Show
        Steve Loughran added a comment - -that would imply passing up machine metadata: cpu family/version, OS, etc. No reason why that couldn't be done, though you'd have to decide whether that is something you'd republish every heartbeat or just when the TT first registers. Of course, without the JT making decisions on where to route stuff based on those features, it's wasted effort. Which would imply you also need some plugin support for making the decisions as to where to run Mappers and Reducers; right now it's fairly straightforward: do it close to the data.
        Hide
        Elton Tian added a comment -

        I like the idea, but I don't think we can set the slots with hardware parameters, rather it's application dependant. For example, you have a Quad core cluster and a Dual core cluster. Both cluster have same disk and inter connection. When you run a "Grep", if you apply the same slot numbers on both cluster, I guess the processing times are similar. If you change you application to "Sort", still using same number of slots, then there could be noticeable difference.

        So I guess, to get a reasonable slots, we need to actually run the application. Somehow.

        Show
        Elton Tian added a comment - I like the idea, but I don't think we can set the slots with hardware parameters, rather it's application dependant. For example, you have a Quad core cluster and a Dual core cluster. Both cluster have same disk and inter connection. When you run a "Grep", if you apply the same slot numbers on both cluster, I guess the processing times are similar. If you change you application to "Sort", still using same number of slots, then there could be noticeable difference. So I guess, to get a reasonable slots, we need to actually run the application. Somehow.
        Hide
        Allen Wittenauer added a comment -

        Closing as won't fix.

        Show
        Allen Wittenauer added a comment - Closing as won't fix.

          People

          • Assignee:
            Unassigned
            Reporter:
            Steve Loughran
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development