Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-2915

Enable YARN RM scale out via federation using multiple RM's

    XMLWordPrintableJSON

    Details

    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      A federation-based approach to transparently scale a single YARN cluster to tens of thousands of nodes, by federating multiple YARN standalone clusters (sub-clusters). The applications running in this federated environment will see a single massive YARN cluster and will be able to schedule tasks on any node of the federated cluster. Under the hood, the federation system will negotiate with sub-clusters ResourceManagers and provide resources to the application. The goal is to allow an individual job to “span” sub-clusters seamlessly.
      Show
      A federation-based approach to transparently scale a single YARN cluster to tens of thousands of nodes, by federating multiple YARN standalone clusters (sub-clusters). The applications running in this federated environment will see a single massive YARN cluster and will be able to schedule tasks on any node of the federated cluster. Under the hood, the federation system will negotiate with sub-clusters ResourceManagers and provide resources to the application. The goal is to allow an individual job to “span” sub-clusters seamlessly.

      Description

      This is an umbrella JIRA that proposes to scale out YARN to support large clusters comprising of tens of thousands of nodes. That is, rather than limiting a YARN managed cluster to about 4k in size, the proposal is to enable the YARN managed cluster to be elastically scalable.

        Attachments

        1. FEDERATION_CAPACITY_ALLOCATION_JIRA.pdf
          751 kB
          Carlo Curino
        2. Federation-BoF.pdf
          909 kB
          Subru Krishnan
        3. federation-prototype.patch
          729 kB
          Subru Krishnan
        4. Yarn_federation_design_v1.pdf
          787 kB
          Subru Krishnan
        5. YARN-Federation-Hadoop-Summit_final.pptx
          182 kB
          Subru Krishnan

          Issue Links

          blocks

          Sub-task - The sub-task of the issue YARN-6848 Move Router ClientRMServices Interceptor and chain into yarn api and common package

          • Major - Major loss of function.
          • Open
          depends upon

          Improvement - An improvement or enhancement to an existing feature or task. YARN-4879 Enhance Allocate Protocol to Identify Requests Explicitly

          • Major - Major loss of function.
          • Resolved
          is blocked by

          Bug - A problem which impairs or prevents the functions of the product. YETUS-485 Yetus run is failing on branch after rebase/force push

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          is depended upon by

          New Feature - A new feature of the product, which has yet to be developed. REEF-337 Support REEF on YARN Federation

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. YARN-5597 YARN Federation improvements

          • Major - Major loss of function.
          • Open
          is related to

          Sub-task - The sub-task of the issue YARN-5357 Timeline service v2 integration with Federation

          • Major - Major loss of function.
          • Open

          Sub-task - The sub-task of the issue YARN-2884 Proxying all AM-RM communications

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-12427 [JDK8] Upgrade Mockito version to 1.10.19

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          relates to

          New Feature - A new feature of the product, which has yet to be developed. HADOOP-13378 Common features between YARN and HDFS Router-based federation

          • Major - Major loss of function.
          • Open

          New Feature - A new feature of the product, which has yet to be developed. HDFS-10467 Router-based HDFS federation

          • Major - Major loss of function.
          • Resolved

          Task - A task that needs to be done. REEF-568 Work around the federated YARN node reports problem

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Task - A task that needs to be done. REEF-589 REEF crashes when new nodes are added to the clusters dynamically

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          There are no Sub-Tasks for this issue.

            Activity

              People

              • Assignee:
                subru Subru Krishnan
                Reporter:
                sriramsrao Sriram Rao
              • Votes:
                2 Vote for this issue
                Watchers:
                90 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: