Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-3983

Make CapacityScheduler to easier extend application allocation logic

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      While working on YARN-1651 (resource allocation for increasing container), I found it is very hard to extend existing CapacityScheduler resource allocation logic to support different types of resource allocation.

      For example, there's a lot of differences between increasing a container and allocating a container:

      • Increasing a container doesn't need to check locality delay.
      • Increasing a container doesn't need to build/modify a resource request tree (ANY->RACK/HOST).
      • Increasing a container doesn't need to check allocation/reservation starvation (see shouldAllocOrReserveNewContainer).
      • After increasing a container is approved by scheduler, it need to update an existing container token instead of creating new container.

      And there're lots of similarities when allocating different types of resources.

      • User-limit/queue-limit will be enforced for both of them.
      • Both of them needs resource reservation logic. (Maybe continuous reservation looking is needed for both of them).

      The purpose of this JIRA is to make easier extending CapacityScheduler resource allocation logic to support different types of resource allocation, make common code reusable, and also better code organization.

        Attachments

        1. YARN-3983.4.patch
          99 kB
          Wangda Tan
        2. YARN-3983.3.patch
          98 kB
          Wangda Tan
        3. YARN-3983.2.patch
          93 kB
          Wangda Tan
        4. YARN-3983.1.patch
          93 kB
          Wangda Tan

          Issue Links

            Activity

              People

              • Assignee:
                leftnoteasy Wangda Tan
                Reporter:
                leftnoteasy Wangda Tan
              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: