Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-6050

AMs can't be scheduled on racks or nodes

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.9.0, 3.0.0-alpha2
    • Fix Version/s: 2.9.0, 3.0.0-alpha4
    • Component/s: None
    • Labels:
      None

      Description

      Yarn itself supports rack/node aware scheduling for AMs; however, there currently are two problems:

      1. To specify hard or soft rack/node requests, you have to specify more than one ResourceRequest. For example, if you want to schedule an AM only on "rackA", you have to create two ResourceRequest, like this:
        ResourceRequest.newInstance(PRIORITY, ANY, CAPABILITY, NUM_CONTAINERS, false);
        ResourceRequest.newInstance(PRIORITY, "rackA", CAPABILITY, NUM_CONTAINERS, true);
        

        The problem is that the Yarn API doesn't actually allow you to specify more than one ResourceRequest in the ApplicationSubmissionContext. The current behavior is to either build one from getResource or directly from getAMContainerResourceRequest, depending on if getAMContainerResourceRequest is null or not. We'll need to add a third method, say getAMContainerResourceRequests, which takes a list of ResourceRequest so that clients can specify the multiple resource requests.

      2. There are some places where things are hardcoded to overwrite what the client specifies. These are pretty straightforward to fix.

        Attachments

        1. YARN-6050.001.patch
          55 kB
          Robert Kanter
        2. YARN-6050.002.patch
          62 kB
          Robert Kanter
        3. YARN-6050.003.patch
          69 kB
          Robert Kanter
        4. YARN-6050.004.patch
          69 kB
          Robert Kanter
        5. YARN-6050.005.patch
          70 kB
          Robert Kanter
        6. YARN-6050.006.patch
          71 kB
          Robert Kanter
        7. YARN-6050.007.patch
          91 kB
          Robert Kanter
        8. YARN-6050.008.patch
          104 kB
          Robert Kanter
        9. YARN-6050.009.patch
          116 kB
          Robert Kanter
        10. YARN-6050.010.patch
          117 kB
          Robert Kanter
        11. YARN-6050.011.patch
          117 kB
          Robert Kanter
        12. YARN-6050.012.patch
          108 kB
          Robert Kanter
        13. YARN-6050.013.patch
          110 kB
          Robert Kanter
        14. YARN-6050.014.branch-2.patch
          109 kB
          Robert Kanter
        15. YARN-6050.014.patch
          110 kB
          Robert Kanter

          Issue Links

            Activity

              People

              • Assignee:
                rkanter Robert Kanter
                Reporter:
                rkanter Robert Kanter
              • Votes:
                0 Vote for this issue
                Watchers:
                11 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: