Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6871

Allow users to specify racks and nodes for strict locality for AMs

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-alpha4
    • Component/s: client
    • Labels:
      None

      Description

      YARN-6050 fixed the YARN API to allow multiple ResourceRequest's when submitting an AM so that you can actually do rack or node locality. We should allow MapReduce users to take advantage of this by exposing this functionality in some way. The raw YARN API allows for a lot of flexibility (e.g. different resources per request, etc), but we don't necessarily want to allow the user to do too much here so they don't shoot themselves in the foot and we don't make this overly complicated.

      I propose we allow users to specify racks and nodes for strict locality. This would allow users to restrict an MR AM to specific racks and/or nodes. We could add a new property, mapreduce.job.am.resource-request.strict.locality, which takes a comma-separated list of entries like:

      • /<rack>
      • /<rack>/<node>
      • <node> (assumes /default-rack)

      MapReduce would then use this information to create the corresponding ResourceRequest's.

      For example, mapreduce.job.am.resource-request.strict.locality=/rack1/node1 would create the following ResourceRequest's:

      • resourceName=ANY, relaxLocality=false, capability=<X,Y>
      • resourceName=/rack1, relaxLocality=false, capability=<X,Y>
      • resourceName=node1, relaxLocality=true, capability=<X,Y>

      By default, the property would be unset, and you'd get the normal ANY ResourceRequest.

        Attachments

        1. MAPREDUCE-6871.001.patch
          14 kB
          Robert Kanter
        2. MAPREDUCE-6871.002.patch
          16 kB
          Robert Kanter
        3. MAPREDUCE-6871.003.patch
          18 kB
          Robert Kanter
        4. MAPREDUCE-6871.004.patch
          17 kB
          Robert Kanter
        5. MAPREDUCE-6871.005.patch
          17 kB
          Robert Kanter
        6. MAPREDUCE-6871.005.patch
          17 kB
          Robert Kanter

          Issue Links

            Activity

              People

              • Assignee:
                rkanter Robert Kanter
                Reporter:
                rkanter Robert Kanter
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: