Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-6344

Add parameter for rack locality delay in CapacityScheduler

    XMLWordPrintableJSON

    Details

    • Hadoop Flags:
      Reviewed

      Description

      When relaxing locality from node to rack, the node-locality-parameter is used: when scheduling opportunities for a scheduler key are more than the value of this parameter, we relax locality and try to assign the container to a node in the corresponding rack.

      On the other hand, when relaxing locality to off-switch (i.e., assign the container anywhere in the cluster), we are using a localityWaitFactor, which is computed based on the number of outstanding requests for a specific scheduler key, which is divided by the size of the cluster.
      In case of applications that request containers in big batches (e.g., traditional MR jobs), and for relatively small clusters, the localityWaitFactor does not affect relaxing locality much.
      However, in case of applications that request containers in small batches, this load factor takes a very small value, which leads to assigning off-switch containers too soon. This situation is even more pronounced in big clusters.
      For example, if an application requests only one container per request, the locality will be relaxed after a single missed scheduling opportunity.

      The purpose of this JIRA is to rethink the way we are relaxing locality for off-switch assignments.

        Attachments

        1. YARN-6344-branch-2.8.patch
          20 kB
          Konstantinos Karanasos
        2. YARN-6344.004.patch
          21 kB
          Konstantinos Karanasos
        3. YARN-6344.003.patch
          31 kB
          Konstantinos Karanasos
        4. YARN-6344.002.patch
          31 kB
          Konstantinos Karanasos
        5. YARN-6344.001.patch
          27 kB
          Konstantinos Karanasos

          Issue Links

            Activity

              People

              • Assignee:
                kkaranasos Konstantinos Karanasos
                Reporter:
                kkaranasos Konstantinos Karanasos
              • Votes:
                0 Vote for this issue
                Watchers:
                11 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: