Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-45 [Preemption] Scheduler feedback to AM to release containers
  3. YARN-2022

Preempting an Application Master container can be kept as least priority when multiple applications are marked for preemption by ProportionalCapacityPreemptionPolicy

    Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.4.0
    • Fix Version/s: 2.5.0
    • Component/s: resourcemanager
    • Labels:
      None

      Description

      Cluster Size = 16GB [2NM's]
      Queue A Capacity = 50%
      Queue B Capacity = 50%
      Consider there are 3 applications running in Queue A which has taken the full cluster capacity.
      J1 = 2GB AM + 1GB * 4 Maps
      J2 = 2GB AM + 1GB * 4 Maps
      J3 = 2GB AM + 1GB * 2 Maps

      Another Job J4 is submitted in Queue B [J4 needs a 2GB AM + 1GB * 2 Maps ].
      Currently in this scenario, Jobs J3 will get killed including its AM.

      It is better if AM can be given least priority among multiple applications. In this same scenario, map tasks from J3 and J2 can be preempted.
      Later when cluster is free, maps can be allocated to these Jobs.

        Attachments

        1. Yarn-2022.1.patch
          10 kB
          Sunil Govindan
        2. YARN-2022-DesignDraft.docx
          32 kB
          Sunil Govindan
        3. YARN-2022.2.patch
          13 kB
          Sunil Govindan
        4. YARN-2022.3.patch
          14 kB
          Sunil Govindan
        5. YARN-2022.4.patch
          15 kB
          Sunil Govindan
        6. YARN-2022.5.patch
          15 kB
          Sunil Govindan
        7. YARN-2022.6.patch
          16 kB
          Sunil Govindan
        8. YARN-2022.7.patch
          21 kB
          Sunil Govindan
        9. YARN-2022.8.patch
          21 kB
          Sunil Govindan
        10. YARN-2022.9.patch
          23 kB
          Sunil Govindan
        11. YARN-2022.10.patch
          23 kB
          Sunil Govindan

          Issue Links

            Activity

              People

              • Assignee:
                sunilg Sunil Govindan
                Reporter:
                sunilg Sunil Govindan
              • Votes:
                0 Vote for this issue
                Watchers:
                15 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: