Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-2915 Enable YARN RM scale out via federation using multiple RM's
  3. YARN-5407

In-memory based implementation of the FederationApplicationStateStore, FederationPolicyStateStore

Details

    • Reviewed

    Description

      YARN-5307 defines the FederationApplicationStateStore API. YARN-3664 defines the FederationPolicyStateStore API. This JIRA tracks an in-memory based implementation which is useful for both single-box testing and for future unit tests that depend on the state store.

      Attachments

        1. YARN-5407-YARN-2915.v0.patch
          29 kB
          Ellen Hui
        2. YARN-5407-YARN-2915.v1.patch
          37 kB
          Ellen Hui
        3. YARN-5407-YARN-2915.v2.patch
          45 kB
          Ellen Hui

        Issue Links

          Activity

            hadoopqa Hadoop QA added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 13s Docker mode activated.
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
            +1 mvninstall 8m 40s YARN-2915 passed
            +1 compile 0m 23s YARN-2915 passed
            +1 checkstyle 0m 15s YARN-2915 passed
            +1 mvnsite 0m 25s YARN-2915 passed
            +1 mvneclipse 0m 13s YARN-2915 passed
            +1 findbugs 0m 51s YARN-2915 passed
            +1 javadoc 0m 18s YARN-2915 passed
            +1 mvninstall 0m 24s the patch passed
            +1 compile 0m 20s the patch passed
            +1 javac 0m 20s the patch passed
            -1 checkstyle 0m 11s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common: The patch generated 4 new + 0 unchanged - 0 fixed = 4 total (was 0)
            +1 mvnsite 0m 25s the patch passed
            +1 mvneclipse 0m 10s the patch passed
            +1 whitespace 0m 0s The patch has no whitespace issues.
            +1 findbugs 0m 56s the patch passed
            +1 javadoc 0m 16s the patch passed
            +1 unit 0m 34s hadoop-yarn-server-common in the patch passed.
            +1 asflicense 0m 20s The patch does not generate ASF License warnings.
            15m 36s



            Subsystem Report/Notes
            Docker Image:yetus/hadoop:9560f25
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822390/YARN-5407-YARN-2915.v0.patch
            JIRA Issue YARN-5407
            Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
            uname Linux 7730734b8100 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
            git revision YARN-2915 / a6a43c0
            Default Java 1.8.0_101
            findbugs v3.0.0
            checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12662/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt
            Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12662/testReport/
            modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
            Console output https://builds.apache.org/job/PreCommit-YARN-Build/12662/console
            Powered by Apache Yetus 0.3.0 http://yetus.apache.org

            This message was automatically generated.

            hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 8m 40s YARN-2915 passed +1 compile 0m 23s YARN-2915 passed +1 checkstyle 0m 15s YARN-2915 passed +1 mvnsite 0m 25s YARN-2915 passed +1 mvneclipse 0m 13s YARN-2915 passed +1 findbugs 0m 51s YARN-2915 passed +1 javadoc 0m 18s YARN-2915 passed +1 mvninstall 0m 24s the patch passed +1 compile 0m 20s the patch passed +1 javac 0m 20s the patch passed -1 checkstyle 0m 11s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common: The patch generated 4 new + 0 unchanged - 0 fixed = 4 total (was 0) +1 mvnsite 0m 25s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 56s the patch passed +1 javadoc 0m 16s the patch passed +1 unit 0m 34s hadoop-yarn-server-common in the patch passed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 15m 36s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822390/YARN-5407-YARN-2915.v0.patch JIRA Issue YARN-5407 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 7730734b8100 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision YARN-2915 / a6a43c0 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12662/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12662/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common Console output https://builds.apache.org/job/PreCommit-YARN-Build/12662/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
            hadoopqa Hadoop QA added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 13s Docker mode activated.
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
            +1 mvninstall 10m 14s YARN-2915 passed
            +1 compile 0m 20s YARN-2915 passed
            +1 checkstyle 0m 14s YARN-2915 passed
            +1 mvnsite 0m 25s YARN-2915 passed
            +1 mvneclipse 0m 14s YARN-2915 passed
            +1 findbugs 0m 47s YARN-2915 passed
            +1 javadoc 0m 17s YARN-2915 passed
            +1 mvninstall 0m 20s the patch passed
            +1 compile 0m 18s the patch passed
            +1 javac 0m 18s the patch passed
            -1 checkstyle 0m 11s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common: The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
            +1 mvnsite 0m 25s the patch passed
            +1 mvneclipse 0m 10s the patch passed
            +1 whitespace 0m 0s The patch has no whitespace issues.
            +1 findbugs 0m 48s the patch passed
            +1 javadoc 0m 13s the patch passed
            +1 unit 0m 30s hadoop-yarn-server-common in the patch passed.
            +1 asflicense 0m 15s The patch does not generate ASF License warnings.
            16m 33s



            Subsystem Report/Notes
            Docker Image:yetus/hadoop:9560f25
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822683/YARN-5407-YARN-2915.v1.patch
            JIRA Issue YARN-5407
            Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
            uname Linux 4f0a542d82d9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
            git revision YARN-2915 / dbaebf8
            Default Java 1.8.0_101
            findbugs v3.0.0
            checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12681/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt
            Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12681/testReport/
            modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
            Console output https://builds.apache.org/job/PreCommit-YARN-Build/12681/console
            Powered by Apache Yetus 0.3.0 http://yetus.apache.org

            This message was automatically generated.

            hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 10m 14s YARN-2915 passed +1 compile 0m 20s YARN-2915 passed +1 checkstyle 0m 14s YARN-2915 passed +1 mvnsite 0m 25s YARN-2915 passed +1 mvneclipse 0m 14s YARN-2915 passed +1 findbugs 0m 47s YARN-2915 passed +1 javadoc 0m 17s YARN-2915 passed +1 mvninstall 0m 20s the patch passed +1 compile 0m 18s the patch passed +1 javac 0m 18s the patch passed -1 checkstyle 0m 11s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common: The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) +1 mvnsite 0m 25s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 48s the patch passed +1 javadoc 0m 13s the patch passed +1 unit 0m 30s hadoop-yarn-server-common in the patch passed. +1 asflicense 0m 15s The patch does not generate ASF License warnings. 16m 33s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822683/YARN-5407-YARN-2915.v1.patch JIRA Issue YARN-5407 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 4f0a542d82d9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision YARN-2915 / dbaebf8 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12681/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12681/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common Console output https://builds.apache.org/job/PreCommit-YARN-Build/12681/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
            subru Subramaniam Krishnan added a comment - - edited

            Thanks ellenfkh for the patch.

            I realized while reviewing this patch that SubClusterPolicyConfiguration is missing queue field. Consequently we can remove it from SetSubClusterPolicyConfigurationRequest. Can you kindly make the changes?

            The patch itself LGTM. I have a minor suggestion for FederationStateStoreBaseTest - we should have helper methods to add applicationSubCluster mapping and policy similar to queryApplicationHomeSC/queryPolicy as they are used by all other tests and now we have repeating code.

            jianhe, can you take a quick look?

            subru Subramaniam Krishnan added a comment - - edited Thanks ellenfkh for the patch. I realized while reviewing this patch that SubClusterPolicyConfiguration is missing queue field. Consequently we can remove it from SetSubClusterPolicyConfigurationRequest . Can you kindly make the changes? The patch itself LGTM. I have a minor suggestion for FederationStateStoreBaseTest - we should have helper methods to add applicationSubCluster mapping and policy similar to queryApplicationHomeSC/queryPolicy as they are used by all other tests and now we have repeating code. jianhe , can you take a quick look?
            jianhe Jian He added a comment -

            lgtm

            jianhe Jian He added a comment - lgtm
            ellenfkh Ellen Hui added a comment -

            Address feedback from subru

            • Move queue from SetSubClusterPolicyConfigurationRequest to SubClusterPolicyConfiguration
            • Add helper methods for registerSubCluster/addApplication/setPolicy in FederationStateStoreBaseTest
            ellenfkh Ellen Hui added a comment - Address feedback from subru Move queue from SetSubClusterPolicyConfigurationRequest to SubClusterPolicyConfiguration Add helper methods for registerSubCluster/addApplication/setPolicy in FederationStateStoreBaseTest
            hadoopqa Hadoop QA added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 0s Docker mode activated.
            -1 patch 0m 6s YARN-5407 does not apply to YARN-2915. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



            This message was automatically generated.

            hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 6s YARN-5407 does not apply to YARN-2915 . Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822882/YARN-5408-YARN-2915.v2.patch JIRA Issue YARN-5407 Console output https://builds.apache.org/job/PreCommit-YARN-Build/12701/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
            ellenfkh Ellen Hui added a comment -

            Wrong diff uploaded previously

            ellenfkh Ellen Hui added a comment - Wrong diff uploaded previously
            hadoopqa Hadoop QA added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 15s Docker mode activated.
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
            +1 mvninstall 8m 9s YARN-2915 passed
            +1 compile 0m 23s YARN-2915 passed
            +1 checkstyle 0m 15s YARN-2915 passed
            +1 mvnsite 0m 29s YARN-2915 passed
            +1 mvneclipse 0m 15s YARN-2915 passed
            +1 findbugs 0m 59s YARN-2915 passed
            +1 javadoc 0m 18s YARN-2915 passed
            +1 mvninstall 0m 25s the patch passed
            +1 compile 0m 23s the patch passed
            +1 cc 0m 23s the patch passed
            +1 javac 0m 23s the patch passed
            -1 checkstyle 0m 11s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common: The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
            +1 mvnsite 0m 29s the patch passed
            +1 mvneclipse 0m 13s the patch passed
            +1 whitespace 0m 0s The patch has no whitespace issues.
            +1 findbugs 1m 0s the patch passed
            +1 javadoc 0m 19s the patch passed
            +1 unit 0m 38s hadoop-yarn-server-common in the patch passed.
            +1 asflicense 0m 17s The patch does not generate ASF License warnings.
            15m 45s



            Subsystem Report/Notes
            Docker Image:yetus/hadoop:9560f25
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822884/YARN-5407-YARN-2915.v2.patch
            JIRA Issue YARN-5407
            Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
            uname Linux 07cdcfbc67a7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
            git revision YARN-2915 / dbaebf8
            Default Java 1.8.0_101
            findbugs v3.0.0
            checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12702/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt
            Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12702/testReport/
            modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
            Console output https://builds.apache.org/job/PreCommit-YARN-Build/12702/console
            Powered by Apache Yetus 0.3.0 http://yetus.apache.org

            This message was automatically generated.

            hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 8m 9s YARN-2915 passed +1 compile 0m 23s YARN-2915 passed +1 checkstyle 0m 15s YARN-2915 passed +1 mvnsite 0m 29s YARN-2915 passed +1 mvneclipse 0m 15s YARN-2915 passed +1 findbugs 0m 59s YARN-2915 passed +1 javadoc 0m 18s YARN-2915 passed +1 mvninstall 0m 25s the patch passed +1 compile 0m 23s the patch passed +1 cc 0m 23s the patch passed +1 javac 0m 23s the patch passed -1 checkstyle 0m 11s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common: The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) +1 mvnsite 0m 29s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 0s the patch passed +1 javadoc 0m 19s the patch passed +1 unit 0m 38s hadoop-yarn-server-common in the patch passed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 15m 45s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822884/YARN-5407-YARN-2915.v2.patch JIRA Issue YARN-5407 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux 07cdcfbc67a7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision YARN-2915 / dbaebf8 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12702/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12702/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common Console output https://builds.apache.org/job/PreCommit-YARN-Build/12702/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.

            +1 on the latest patch, I committed to branch YARN-2915.

            Thanks ellenfkh for working on this & jianhe for reviewing it.

            subru Subramaniam Krishnan added a comment - +1 on the latest patch, I committed to branch YARN-2915 . Thanks ellenfkh for working on this & jianhe for reviewing it.

            People

              ellenfkh Ellen Hui
              subru Subramaniam Krishnan
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: