Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-45 [Preemption] Scheduler feedback to AM to release containers
  3. YARN-2022

Preempting an Application Master container can be kept as least priority when multiple applications are marked for preemption by ProportionalCapacityPreemptionPolicy

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 2.4.0
    • 2.5.0
    • resourcemanager
    • None

    Description

      Cluster Size = 16GB [2NM's]
      Queue A Capacity = 50%
      Queue B Capacity = 50%
      Consider there are 3 applications running in Queue A which has taken the full cluster capacity.
      J1 = 2GB AM + 1GB * 4 Maps
      J2 = 2GB AM + 1GB * 4 Maps
      J3 = 2GB AM + 1GB * 2 Maps

      Another Job J4 is submitted in Queue B [J4 needs a 2GB AM + 1GB * 2 Maps ].
      Currently in this scenario, Jobs J3 will get killed including its AM.

      It is better if AM can be given least priority among multiple applications. In this same scenario, map tasks from J3 and J2 can be preempted.
      Later when cluster is free, maps can be allocated to these Jobs.

      Attachments

        1. Yarn-2022.1.patch
          10 kB
          Sunil G
        2. YARN-2022.10.patch
          23 kB
          Sunil G
        3. YARN-2022.2.patch
          13 kB
          Sunil G
        4. YARN-2022.3.patch
          14 kB
          Sunil G
        5. YARN-2022.4.patch
          15 kB
          Sunil G
        6. YARN-2022.5.patch
          15 kB
          Sunil G
        7. YARN-2022.6.patch
          16 kB
          Sunil G
        8. YARN-2022.7.patch
          21 kB
          Sunil G
        9. YARN-2022.8.patch
          21 kB
          Sunil G
        10. YARN-2022.9.patch
          23 kB
          Sunil G
        11. YARN-2022-DesignDraft.docx
          32 kB
          Sunil G

        Issue Links

          Activity

            People

              sunilg Sunil G
              sunilg Sunil G
              Votes:
              0 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: