Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-4120

FSAppAttempt.getResourceUsage() should not take preemptedResource into account

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: fairscheduler
    • Labels:
      None

      Description

      When compute resource usage for Schedulables, the following code is envolved,
      FSAppAttempt.getResourceUsage,

      public Resource getResourceUsage() {
        return Resources.subtract(getCurrentConsumption(), getPreemptedResources());
      }
      

      and this value is aggregated to FSLeafQueues and FSParentQueues. In my opinion, taking preemptedResource into account here is not reasonable, there are two main reasons,

      1. it is something in future, i.e., even though these resources are marked as preempted, it is currently used by app, and these resources will be subtracted from currentCosumption once the preemption is finished. it's not reasonable to make arrange for it ahead of time.
      2. there's another problem here, consider following case,
                    root
                   /    \
              queue1   queue2
              /    \
        queue1.3, queue1.4
        

        suppose queue1.3 need resource and it can preempt resources from queue1.4, the preemption happens in the interior of queue1. But when compute resource usage of queue1, queue1.resourceUsage = it's_current_resource_usage - preemption according to the current code, which is unfair to queue2 when doing resource allocating.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                xinxianyin Xianyin Xin
              • Votes:
                0 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: