Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-7655

Avoid AM preemption caused by RRs for specific nodes or racks

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0
    • Fix Version/s: 3.1.0
    • Component/s: fairscheduler
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      We frequently see AM preemptions when starvedApp.getStarvedResourceRequests() in FSPreemptionThread#identifyContainersToPreempt includes one or more RRs that request containers on a specific node. Since this causes us to only consider one node to preempt containers on, the really good work that was done in YARN-5830 doesn't save us from AM preemption. Even though there might be multiple nodes on which we could preempt enough non-AM containers to satisfy the app's starvation, we often wind up preempting one or more AM containers on the single node that we're considering.

      A proposed solution is that if we're going to preempt one or more AM containers for an RR that specifies a node or rack, then we should instead expand the search space to consider all nodes. That way we take advantage of YARN-5830, and only preempt AMs if there's no alternative. I've attached a patch with an initial implementation of this. We've been running it on a few clusters, and have seen AM preemptions drop from double-digit occurrences on many days to zero.

      Of course, the tradeoff is some loss of locality, since the starved app is less likely to be allocated resources at the most specific locality level that it asked for. My opinion is that this tradeoff is worth it, but interested to hear what others think as well.

        Attachments

        1. YARN-7655-001.patch
          8 kB
          Steven Rand
        2. YARN-7655-002.patch
          9 kB
          Steven Rand
        3. YARN-7655-003.patch
          9 kB
          Steven Rand
        4. YARN-7655-004.patch
          9 kB
          Steven Rand

          Issue Links

            Activity

              People

              • Assignee:
                Steven Rand Steven Rand
                Reporter:
                Steven Rand Steven Rand
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: