Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-10613

Config to allow Intra- and Inter-queue preemption to enable/disable conservativeDRF

    XMLWordPrintableJSON

Details

    Description

      YARN-8292 added code that prevents CS intra-queue preemption from preempting containers from an app unless all of the major resources used by the app are greater than the user limit for that user.

      Ex:

      Used User Limit
      <58GB, 58> <30GB, 300>

      In this example, only used memory is above the user limit, not used vcores. So, intra-queue preemption will not occur.

      YARN-8292 added the conservativeDRF flag to CapacitySchedulerPreemptionUtils#tryPreemptContainerAndDeductResToObtain. If conservativeDRF is false, containers will be preempted from apps in the example state. If true, containers will not be preempted.

      This flag is hard-coded to false for Inter-queue (cross-queue) preemption and true for intra-queue (in-queue) preemption.

      I propose that in some cases, we want intra-queue preemption to be more aggressive and preempt in the example case. To accommodate that, I propose the addition of a config property.

      Also, we may want inter-queue (cross-queue) preemption to be more conservative, so I propose also making that a configuration property:

      Attachments

        1. YARN-10613.branch-2.10.002.patch
          20 kB
          Eric Payne
        2. YARN-10613.branch-3.2.002.patch
          20 kB
          Eric Payne
        3. YARN-10613.trunk.001.patch
          20 kB
          Eric Payne
        4. YARN-10613.trunk.002.patch
          20 kB
          Eric Payne

        Activity

          People

            epayne Eric Payne
            epayne Eric Payne
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: