Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-6050

AMs can't be scheduled on racks or nodes

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.9.0, 3.0.0-alpha2
    • 2.9.0, 3.0.0-alpha4
    • None
    • None

    Description

      Yarn itself supports rack/node aware scheduling for AMs; however, there currently are two problems:

      1. To specify hard or soft rack/node requests, you have to specify more than one ResourceRequest. For example, if you want to schedule an AM only on "rackA", you have to create two ResourceRequest, like this:
        ResourceRequest.newInstance(PRIORITY, ANY, CAPABILITY, NUM_CONTAINERS, false);
        ResourceRequest.newInstance(PRIORITY, "rackA", CAPABILITY, NUM_CONTAINERS, true);
        

        The problem is that the Yarn API doesn't actually allow you to specify more than one ResourceRequest in the ApplicationSubmissionContext. The current behavior is to either build one from getResource or directly from getAMContainerResourceRequest, depending on if getAMContainerResourceRequest is null or not. We'll need to add a third method, say getAMContainerResourceRequests, which takes a list of ResourceRequest so that clients can specify the multiple resource requests.

      2. There are some places where things are hardcoded to overwrite what the client specifies. These are pretty straightforward to fix.

      Attachments

        1. YARN-6050.001.patch
          55 kB
          Robert Kanter
        2. YARN-6050.002.patch
          62 kB
          Robert Kanter
        3. YARN-6050.003.patch
          69 kB
          Robert Kanter
        4. YARN-6050.004.patch
          69 kB
          Robert Kanter
        5. YARN-6050.005.patch
          70 kB
          Robert Kanter
        6. YARN-6050.006.patch
          71 kB
          Robert Kanter
        7. YARN-6050.007.patch
          91 kB
          Robert Kanter
        8. YARN-6050.008.patch
          104 kB
          Robert Kanter
        9. YARN-6050.009.patch
          116 kB
          Robert Kanter
        10. YARN-6050.010.patch
          117 kB
          Robert Kanter
        11. YARN-6050.011.patch
          117 kB
          Robert Kanter
        12. YARN-6050.012.patch
          108 kB
          Robert Kanter
        13. YARN-6050.013.patch
          110 kB
          Robert Kanter
        14. YARN-6050.014.patch
          110 kB
          Robert Kanter
        15. YARN-6050.014.branch-2.patch
          109 kB
          Robert Kanter

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            rkanter Robert Kanter
            rkanter Robert Kanter
            Votes:
            0 Vote for this issue
            Watchers:
            11 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment