Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-2915

Enable YARN RM scale out via federation using multiple RM's

    Details

    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      A federation-based approach to transparently scale a single YARN cluster to tens of thousands of nodes, by federating multiple YARN standalone clusters (sub-clusters). The applications running in this federated environment will see a single massive YARN cluster and will be able to schedule tasks on any node of the federated cluster. Under the hood, the federation system will negotiate with sub-clusters ResourceManagers and provide resources to the application. The goal is to allow an individual job to “span” sub-clusters seamlessly.
      Show
      A federation-based approach to transparently scale a single YARN cluster to tens of thousands of nodes, by federating multiple YARN standalone clusters (sub-clusters). The applications running in this federated environment will see a single massive YARN cluster and will be able to schedule tasks on any node of the federated cluster. Under the hood, the federation system will negotiate with sub-clusters ResourceManagers and provide resources to the application. The goal is to allow an individual job to “span” sub-clusters seamlessly.

      Description

      This is an umbrella JIRA that proposes to scale out YARN to support large clusters comprising of tens of thousands of nodes. That is, rather than limiting a YARN managed cluster to about 4k in size, the proposal is to enable the YARN managed cluster to be elastically scalable.

        Attachments

        1. YARN-Federation-Hadoop-Summit_final.pptx
          182 kB
          Subru Krishnan
        2. Federation-BoF.pdf
          909 kB
          Subru Krishnan
        3. federation-prototype.patch
          729 kB
          Subru Krishnan
        4. FEDERATION_CAPACITY_ALLOCATION_JIRA.pdf
          751 kB
          Carlo Curino
        5. Yarn_federation_design_v1.pdf
          787 kB
          Subru Krishnan

          Issue Links

          There are no Sub-Tasks for this issue.

            Activity

              People

              • Assignee:
                subru Subru Krishnan
                Reporter:
                sriramsrao Sriram Rao
              • Votes:
                2 Vote for this issue
                Watchers:
                81 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: