Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-5035

FairScheduler: Adjust maxAssign dynamically when assignMultiple is turned on

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.8.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: fairscheduler
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Hide
      Introducing a new configuration "yarn.scheduler.fair.dynamic.max.assign" to dynamically determine the resources to assign per heartbeat when assignmultiple is turned on. When turned on, the scheduler allocates roughly half of the remaining resources overriding any max.assign settings configured. This is turned ON by default.
      Show
      Introducing a new configuration "yarn.scheduler.fair.dynamic.max.assign" to dynamically determine the resources to assign per heartbeat when assignmultiple is turned on. When turned on, the scheduler allocates roughly half of the remaining resources overriding any max.assign settings configured. This is turned ON by default.

      Description

      When assignMultiple is turned on, today the number of containers allocated to node within a single heartbeat is determined by maxAssign. Picking a higher value for this prevents the spreading of the workload, and a smaller number leads to slow ramp up.

      The proposal is to make it dynamic - half of the unallocated resources on the node.

        Attachments

        1. yarn-5035-3.patch
          13 kB
          Karthik Kambatla
        2. yarn-5035-2.patch
          13 kB
          Karthik Kambatla
        3. yarn-5035-1.patch
          12 kB
          Karthik Kambatla

          Issue Links

            Activity

              People

              • Assignee:
                kasha Karthik Kambatla
                Reporter:
                kasha Karthik Kambatla
              • Votes:
                0 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: