Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-4287

Capacity Scheduler: Rack Locality improvement

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.7.1
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: capacityscheduler
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      YARN-4189 does an excellent job describing the issues with the current delay scheduling algorithms within the capacity scheduler. The design proposal also seems like a good direction.

      This jira proposes a simple interim solution to the key issue we've been experiencing on a regular basis:

      • rackLocal assignments trickle out due to nodeLocalityDelay. This can have significant impact on things like CombineFileInputFormat which targets very specific nodes in its split calculations.

      I'm not sure when YARN-4189 will become reality so I thought a simple interim patch might make sense. The basic idea is simple:
      1) Separate delays for rackLocal, and OffSwitch (today there is only 1)
      2) When we're getting rackLocal assignments, subsequent rackLocal assignments should not be delayed

      Patch will be uploaded shortly. No big deal if the consensus is to go straight to YARN-4189.

        Attachments

        1. YARN-4287.patch
          30 kB
          Nathan Roberts
        2. YARN-4287-v2.patch
          37 kB
          Nathan Roberts
        3. YARN-4287-v3.patch
          37 kB
          Nathan Roberts
        4. YARN-4287-v4.patch
          46 kB
          Nathan Roberts
        5. YARN-4287-minimal.patch
          15 kB
          Nathan Roberts
        6. YARN-4287-minimal-v2.patch
          15 kB
          Nathan Roberts
        7. YARN-4287-minimal-v3.patch
          15 kB
          Nathan Roberts
        8. YARN-4287-minimal-v4.patch
          21 kB
          Nathan Roberts
        9. YARN-4287-minimal-v4-branch-2.7.patch
          20 kB
          Nathan Roberts

          Issue Links

            Activity

              People

              • Assignee:
                nroberts Nathan Roberts
                Reporter:
                nroberts Nathan Roberts
              • Votes:
                0 Vote for this issue
                Watchers:
                14 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: