XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: YARN-1011
    • Fix Version/s: YARN-1011
    • Component/s: None
    • Labels:
      None

      Description

      Opportunistic Containers for OverAllocation need to be allocated to pending applications in some fair manner. Rather than evaluating queue and user resource usage (allocated resource usage) and comparing against queue and user limits to decide the allocation, it might make more sense to use a snapshot of actual resource utilization of the queue and user.

      To facilitate this, this JIRA proposes to aggregate per user, per app (and maybe per queue) resource utilization in addition to aggregated Container and Node Utilization and send it along with the NM heartbeat. It should be fairly inexpensive to aggregate - since it can be performed in the same loop of the ContainersMonitorImpl's Monitoring thread.

      A snapshot aggregate can be made every couple of seconds in the RM. This instantaneous resource utilization should be used to decide if Opportunistic containers can be allocated to an App, Queue or User.

        Attachments

        1. YARN-8827-YARN-1011.01.patch
          54 kB
          Arun Suresh
        2. YARN-8827-YARN-1011.02.patch
          55 kB
          Arun Suresh
        3. YARN-8827-YARN-1011.03.patch
          55 kB
          Arun Suresh
        4. YARN-8827-YARN-1011.04.patch
          55 kB
          Arun Suresh
        5. YARN-8827-YARN-1011.05.patch
          56 kB
          Arun Suresh
        6. YARN-8827-YARN-1011.06.patch
          56 kB
          Arun Suresh
        7. YARN-8827-YARN-1011.07.patch
          56 kB
          Arun Suresh

          Issue Links

            Activity

              People

              • Assignee:
                asuresh Arun Suresh
                Reporter:
                asuresh Arun Suresh
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: