Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-2915

Enable YARN RM scale out via federation using multiple RM's

          Details

          • Hadoop Flags:
            Reviewed
          • Release Note:
            Hide
            A federation-based approach to transparently scale a single YARN cluster to tens of thousands of nodes, by federating multiple YARN standalone clusters (sub-clusters). The applications running in this federated environment will see a single massive YARN cluster and will be able to schedule tasks on any node of the federated cluster. Under the hood, the federation system will negotiate with sub-clusters ResourceManagers and provide resources to the application. The goal is to allow an individual job to “span” sub-clusters seamlessly.
            Show
            A federation-based approach to transparently scale a single YARN cluster to tens of thousands of nodes, by federating multiple YARN standalone clusters (sub-clusters). The applications running in this federated environment will see a single massive YARN cluster and will be able to schedule tasks on any node of the federated cluster. Under the hood, the federation system will negotiate with sub-clusters ResourceManagers and provide resources to the application. The goal is to allow an individual job to “span” sub-clusters seamlessly.

            Description

            This is an umbrella JIRA that proposes to scale out YARN to support large clusters comprising of tens of thousands of nodes. That is, rather than limiting a YARN managed cluster to about 4k in size, the proposal is to enable the YARN managed cluster to be elastically scalable.

              Attachments

              1. Yarn_federation_design_v1.pdf
                787 kB
                Subru Krishnan
              2. FEDERATION_CAPACITY_ALLOCATION_JIRA.pdf
                751 kB
                Carlo Curino
              3. federation-prototype.patch
                729 kB
                Subru Krishnan
              4. Federation-BoF.pdf
                909 kB
                Subru Krishnan
              5. YARN-Federation-Hadoop-Summit_final.pptx
                182 kB
                Subru Krishnan

                Issue Links

                blocks

                Sub-task - The sub-task of the issue YARN-6848 Move Router ClientRMServices Interceptor and chain into yarn api and common package

                • Major - Major loss of function.
                • Open
                depends upon

                Improvement - An improvement or enhancement to an existing feature or task. YARN-4879 Enhance Allocate Protocol to Identify Requests Explicitly

                • Major - Major loss of function.
                • Resolved
                is blocked by

                Bug - A problem which impairs or prevents the functions of the product. YETUS-485 Yetus run is failing on branch after rebase/force push

                • Critical - Crashes, loss of data, severe memory leak.
                • Resolved
                is depended upon by

                New Feature - A new feature of the product, which has yet to be developed. REEF-337 Support REEF on YARN Federation

                • Major - Major loss of function.
                • Open

                Improvement - An improvement or enhancement to an existing feature or task. YARN-5597 YARN Federation improvements

                • Major - Major loss of function.
                • Open
                is related to

                Sub-task - The sub-task of the issue YARN-5357 Timeline service v2 integration with Federation

                • Major - Major loss of function.
                • Open

                Sub-task - The sub-task of the issue YARN-2884 Proxying all AM-RM communications

                • Major - Major loss of function.
                • Resolved

                Improvement - An improvement or enhancement to an existing feature or task. HADOOP-12427 [JDK8] Upgrade Mockito version to 1.10.19

                • Minor - Minor loss of function, or other problem where easy workaround is present.
                • Resolved
                relates to

                New Feature - A new feature of the product, which has yet to be developed. HADOOP-13378 Common features between YARN and HDFS Router-based federation

                • Major - Major loss of function.
                • Open

                New Feature - A new feature of the product, which has yet to be developed. HDFS-10467 Router-based HDFS federation

                • Major - Major loss of function.
                • Resolved

                Task - A task that needs to be done. REEF-568 Work around the federated YARN node reports problem

                • Minor - Minor loss of function, or other problem where easy workaround is present.
                • Resolved

                Task - A task that needs to be done. REEF-589 REEF crashes when new nodes are added to the clusters dynamically

                • Minor - Minor loss of function, or other problem where easy workaround is present.
                • Resolved
                There are no Sub-Tasks for this issue.

                  Activity

                    People

                    • Assignee:
                      subru Subru Krishnan
                      Reporter:
                      sriramsrao Sriram Rao
                    • Votes:
                      2 Vote for this issue
                      Watchers:
                      88 Start watching this issue

                      Dates

                      • Created:
                        Updated:
                        Resolved: