Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-9598

Make reservation work well when multi-node enabled

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Patch Available
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: capacityscheduler
    • Labels:
      None

      Description

      This issue is to solve problems about reservation when multi-node enabled:

      1. As discussed in YARN-9576, re-reservation proposal may be always generated on the same node and break the scheduling for this app and later apps. I think re-reservation in unnecessary and we can replace it with LOCALITY_SKIPPED to let scheduler have a chance to look up follow candidates for this app when multi-node enabled.
      2. Scheduler iterates all nodes and try to allocate for reserved container in LeafQueue#allocateFromReservedContainer. Here there are two problems:
        • The node of reserved container should be taken as candidates instead of all nodes when calling FiCaSchedulerApp#assignContainers, otherwise later scheduler may generate a reservation-fulfilled proposal on another node, which will always be rejected in FiCaScheduler#commonCheckContainerAllocation.
        • Assignment returned by FiCaSchedulerApp#assignContainers could never be null even if it's just skipped, it will break the normal scheduling process for this leaf queue because of the if clause in LeafQueue#assignContainers: "if (null != assignment) { return assignment;}"
      3. Nodes which have been reserved should be skipped when iterating candidates in RegularContainerAllocator#allocate, otherwise scheduler may generate allocation or reservation proposal on these node which will always be rejected in FiCaScheduler#commonCheckContainerAllocation.

        Attachments

        1. image-2019-06-10-11-37-44-975.png
          59 kB
          Juanjuan Tian
        2. image-2019-06-10-11-37-43-283.png
          59 kB
          Juanjuan Tian
        3. YARN-9598.001.patch
          13 kB
          Tao Yang

          Issue Links

            Activity

              People

              • Assignee:
                Tao Yang Tao Yang
                Reporter:
                Tao Yang Tao Yang
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated: