Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-19

4-layer topology (with NodeGroup layer) implementation of Container Assignment and Task Scheduling (for YARN)

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      There are several classes in YARN’s container assignment and task scheduling algorithms that related to data locality which were updated to give preference to running a container on the same nodegroup. This section summarized the changes in the patch that provides a new implementation to support a four-layer hierarchy.
      When the ApplicationMaster makes a resource allocation request to the scheduler of ResourceManager, it will add the node group to the list of attributes in the ResourceRequest. The parameters of the resource request will change from <priority, (host, rack, *), memory, #containers> to <priority, (host, nodegroup, rack, *), memory, #containers>.
      After receiving the ResoureRequest the RM scheduler will assign containers for requests in the sequence of data-local, nodegroup-local, rack-local and off-switch.Then, ApplicationMaster schedules tasks on allocated containers in sequence of data- local, nodegroup-local, rack-local and off-switch.
      In terms of code changes made to YARN task scheduling, we updated the class ContainerRequestEvent so that applications can requests for containers can include anodegroup. In RM schedulers, FifoScheduler and CapacityScheduler were updated. For the FifoScheduler, the changes were in the method assignContainers. For the Capacity Scheduler the method assignContainersOnNode in the class of LeafQueue was updated. In both changes a new method, assignNodeGroupLocalContainers() was added in between the assignment data-local and rack-local.

        Attachments

        1. YARN-19-v4.patch
          78 kB
          Wenwu Peng
        2. YARN-19-v3-alpha.patch
          75 kB
          Junping Du
        3. YARN-19-v2.patch
          66 kB
          Junping Du
        4. YARN-19.patch
          37 kB
          Junping Du
        5. MAPREDUCE-4310-v1.patch
          67 kB
          Junping Du
        6. MAPREDUCE-4310.patch
          59 kB
          Junping Du
        7. HADOOP-8475-ContainerAssignmentTaskScheduling-withNodeGroup.patch
          73 kB
          Junping Du

          Issue Links

            Activity

              People

              • Assignee:
                junping_du Junping Du
                Reporter:
                junping_du Junping Du
              • Votes:
                0 Vote for this issue
                Watchers:
                25 Start watching this issue

                Dates

                • Created:
                  Updated: