Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-alpha2
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Currently AppSchedulingInfo simply stores resource request and scheduler make decision according to stored resource request. For example, CS/FS use slightly different approach to get pending resource request and make delay scheduling decision.

      There're several benefits of moving pending resource request data structure to SchedulingPlacementSet

      1) Delay scheduling logic should be agnostic to scheduler, for example CS supports count-based delay and FS supports both of count-based and time-based delay. Ideally scheduler should be able to choose which delay scheduling policy to use.
      2) In addition to 1., YARN-4902 has proposal to support pluggable delay scheduling behavior in addition to locality-based (host->rack->offswitch). Which requires more flexibility.
      3) To make YARN-4902 becomes real, instead of directly adding the new resource request API to client, we can make scheduler to use it internally to make sure it is well defined. And AppSchedulingInfo/SchedulingPlacementSet will be the perfect place to isolate which ResourceRequest implementation to use.
      4) Different scheduling requirement needs different behavior of checking ResourceRequest table.

      This JIRA is the 1st patch of several refactorings. Which moves all ResourceRequest data structure and logics to SchedulingPlacementSet. We need follow changes to make it better structured

      • Make delay scheduling to be a plugin of SchedulingPlacementSet
      • After YARN-4902 get committed, change SchedulingPlacementSet to use YARN-4902 internally.

        Attachments

        1. YARN-5906.1.patch
          39 kB
          Wangda Tan
        2. YARN-5906.2.patch
          39 kB
          Wangda Tan
        3. YARN-5906.3.patch
          41 kB
          Wangda Tan
        4. YARN-5906.4.patch
          41 kB
          Wangda Tan
        5. YARN-5906.5.patch
          41 kB
          Wangda Tan
        6. YARN-5906.branch-2.005.patch
          40 kB
          Wangda Tan

          Activity

            People

            • Assignee:
              leftnoteasy Wangda Tan
              Reporter:
              leftnoteasy Wangda Tan
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: