XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.8.2
    • Fix Version/s: None
    • Component/s: yarn
    • Labels:
      None
    • Environment:
      • Google Compute Engine (Dataproc)
      • Java 8
      • Hadoop 2.8.2 using client-mode graceful decommissioning

      Description

      Proposal design doc with background + details (please comment directly on doc): https://docs.google.com/document/d/1hF2Bod_m7rPgSXlunbWGn1cYi3-L61KvQhPlY9Jk9Hk/edit#heading=h.ab4ufqsj47b7

      tl;dr Right now, DECOMMISSIONING nodes must wait for in-progress applications to complete before shutting down, but they cannot run new containers from those in-progress applications. This is wasteful, particularly in environments where you are billed by resource usage (e.g. EC2).

      Proposal: YARN should schedule containers from in-progress applications on DECOMMISSIONING nodes, but should still avoid scheduling containers from new applications. That will make in-progress applications complete faster and let nodes decommission faster. Overall, this should be cheaper.

      I have a working patch without unit tests that's surprisingly just a few real lines of code (patch 001). If folks are happy with the proposal, I'll write unit tests and also write a patch targeted at trunk.

        Attachments

        1. YARN-8118-branch-2.001.patch
          34 kB
          Karthik Palaniappan

          Activity

            People

            • Assignee:
              Karthik Palaniappan Karthik Palaniappan
              Reporter:
              Karthik Palaniappan Karthik Palaniappan
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated: