Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-2915 Enable YARN RM scale out via federation using multiple RM's
  3. YARN-5407

In-memory based implementation of the FederationApplicationStateStore, FederationPolicyStateStore

    Details

    • Hadoop Flags:
      Reviewed

      Description

      YARN-5307 defines the FederationApplicationStateStore API. YARN-3664 defines the FederationPolicyStateStore API. This JIRA tracks an in-memory based implementation which is useful for both single-box testing and for future unit tests that depend on the state store.

      1. YARN-5407-YARN-2915.v0.patch
        29 kB
        Ellen Hui
      2. YARN-5407-YARN-2915.v1.patch
        37 kB
        Ellen Hui
      3. YARN-5407-YARN-2915.v2.patch
        45 kB
        Ellen Hui

        Issue Links

          Activity

          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 8m 40s YARN-2915 passed
          +1 compile 0m 23s YARN-2915 passed
          +1 checkstyle 0m 15s YARN-2915 passed
          +1 mvnsite 0m 25s YARN-2915 passed
          +1 mvneclipse 0m 13s YARN-2915 passed
          +1 findbugs 0m 51s YARN-2915 passed
          +1 javadoc 0m 18s YARN-2915 passed
          +1 mvninstall 0m 24s the patch passed
          +1 compile 0m 20s the patch passed
          +1 javac 0m 20s the patch passed
          -1 checkstyle 0m 11s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common: The patch generated 4 new + 0 unchanged - 0 fixed = 4 total (was 0)
          +1 mvnsite 0m 25s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 56s the patch passed
          +1 javadoc 0m 16s the patch passed
          +1 unit 0m 34s hadoop-yarn-server-common in the patch passed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          15m 36s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822390/YARN-5407-YARN-2915.v0.patch
          JIRA Issue YARN-5407
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 7730734b8100 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision YARN-2915 / a6a43c0
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12662/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12662/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/12662/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 8m 40s YARN-2915 passed +1 compile 0m 23s YARN-2915 passed +1 checkstyle 0m 15s YARN-2915 passed +1 mvnsite 0m 25s YARN-2915 passed +1 mvneclipse 0m 13s YARN-2915 passed +1 findbugs 0m 51s YARN-2915 passed +1 javadoc 0m 18s YARN-2915 passed +1 mvninstall 0m 24s the patch passed +1 compile 0m 20s the patch passed +1 javac 0m 20s the patch passed -1 checkstyle 0m 11s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common: The patch generated 4 new + 0 unchanged - 0 fixed = 4 total (was 0) +1 mvnsite 0m 25s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 56s the patch passed +1 javadoc 0m 16s the patch passed +1 unit 0m 34s hadoop-yarn-server-common in the patch passed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 15m 36s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822390/YARN-5407-YARN-2915.v0.patch JIRA Issue YARN-5407 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 7730734b8100 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision YARN-2915 / a6a43c0 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12662/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12662/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common Console output https://builds.apache.org/job/PreCommit-YARN-Build/12662/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 10m 14s YARN-2915 passed
          +1 compile 0m 20s YARN-2915 passed
          +1 checkstyle 0m 14s YARN-2915 passed
          +1 mvnsite 0m 25s YARN-2915 passed
          +1 mvneclipse 0m 14s YARN-2915 passed
          +1 findbugs 0m 47s YARN-2915 passed
          +1 javadoc 0m 17s YARN-2915 passed
          +1 mvninstall 0m 20s the patch passed
          +1 compile 0m 18s the patch passed
          +1 javac 0m 18s the patch passed
          -1 checkstyle 0m 11s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common: The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
          +1 mvnsite 0m 25s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 48s the patch passed
          +1 javadoc 0m 13s the patch passed
          +1 unit 0m 30s hadoop-yarn-server-common in the patch passed.
          +1 asflicense 0m 15s The patch does not generate ASF License warnings.
          16m 33s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822683/YARN-5407-YARN-2915.v1.patch
          JIRA Issue YARN-5407
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 4f0a542d82d9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision YARN-2915 / dbaebf8
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12681/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12681/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/12681/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 10m 14s YARN-2915 passed +1 compile 0m 20s YARN-2915 passed +1 checkstyle 0m 14s YARN-2915 passed +1 mvnsite 0m 25s YARN-2915 passed +1 mvneclipse 0m 14s YARN-2915 passed +1 findbugs 0m 47s YARN-2915 passed +1 javadoc 0m 17s YARN-2915 passed +1 mvninstall 0m 20s the patch passed +1 compile 0m 18s the patch passed +1 javac 0m 18s the patch passed -1 checkstyle 0m 11s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common: The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) +1 mvnsite 0m 25s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 48s the patch passed +1 javadoc 0m 13s the patch passed +1 unit 0m 30s hadoop-yarn-server-common in the patch passed. +1 asflicense 0m 15s The patch does not generate ASF License warnings. 16m 33s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822683/YARN-5407-YARN-2915.v1.patch JIRA Issue YARN-5407 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 4f0a542d82d9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision YARN-2915 / dbaebf8 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12681/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12681/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common Console output https://builds.apache.org/job/PreCommit-YARN-Build/12681/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          subru Subru Krishnan added a comment - - edited

          Thanks Ellen Hui for the patch.

          I realized while reviewing this patch that SubClusterPolicyConfiguration is missing queue field. Consequently we can remove it from SetSubClusterPolicyConfigurationRequest. Can you kindly make the changes?

          The patch itself LGTM. I have a minor suggestion for FederationStateStoreBaseTest - we should have helper methods to add applicationSubCluster mapping and policy similar to queryApplicationHomeSC/queryPolicy as they are used by all other tests and now we have repeating code.

          Jian He, can you take a quick look?

          Show
          subru Subru Krishnan added a comment - - edited Thanks Ellen Hui for the patch. I realized while reviewing this patch that SubClusterPolicyConfiguration is missing queue field. Consequently we can remove it from SetSubClusterPolicyConfigurationRequest . Can you kindly make the changes? The patch itself LGTM. I have a minor suggestion for FederationStateStoreBaseTest - we should have helper methods to add applicationSubCluster mapping and policy similar to queryApplicationHomeSC/queryPolicy as they are used by all other tests and now we have repeating code. Jian He , can you take a quick look?
          Hide
          jianhe Jian He added a comment -

          lgtm

          Show
          jianhe Jian He added a comment - lgtm
          Hide
          ellenfkh Ellen Hui added a comment -

          Address feedback from Subru Krishnan

          • Move queue from SetSubClusterPolicyConfigurationRequest to SubClusterPolicyConfiguration
          • Add helper methods for registerSubCluster/addApplication/setPolicy in FederationStateStoreBaseTest
          Show
          ellenfkh Ellen Hui added a comment - Address feedback from Subru Krishnan Move queue from SetSubClusterPolicyConfigurationRequest to SubClusterPolicyConfiguration Add helper methods for registerSubCluster/addApplication/setPolicy in FederationStateStoreBaseTest
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 patch 0m 6s YARN-5407 does not apply to YARN-2915. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          Subsystem Report/Notes
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822882/YARN-5408-YARN-2915.v2.patch
          JIRA Issue YARN-5407
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/12701/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 6s YARN-5407 does not apply to YARN-2915 . Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822882/YARN-5408-YARN-2915.v2.patch JIRA Issue YARN-5407 Console output https://builds.apache.org/job/PreCommit-YARN-Build/12701/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          ellenfkh Ellen Hui added a comment -

          Wrong diff uploaded previously

          Show
          ellenfkh Ellen Hui added a comment - Wrong diff uploaded previously
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 8m 9s YARN-2915 passed
          +1 compile 0m 23s YARN-2915 passed
          +1 checkstyle 0m 15s YARN-2915 passed
          +1 mvnsite 0m 29s YARN-2915 passed
          +1 mvneclipse 0m 15s YARN-2915 passed
          +1 findbugs 0m 59s YARN-2915 passed
          +1 javadoc 0m 18s YARN-2915 passed
          +1 mvninstall 0m 25s the patch passed
          +1 compile 0m 23s the patch passed
          +1 cc 0m 23s the patch passed
          +1 javac 0m 23s the patch passed
          -1 checkstyle 0m 11s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common: The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
          +1 mvnsite 0m 29s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 0s the patch passed
          +1 javadoc 0m 19s the patch passed
          +1 unit 0m 38s hadoop-yarn-server-common in the patch passed.
          +1 asflicense 0m 17s The patch does not generate ASF License warnings.
          15m 45s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822884/YARN-5407-YARN-2915.v2.patch
          JIRA Issue YARN-5407
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux 07cdcfbc67a7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision YARN-2915 / dbaebf8
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12702/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12702/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/12702/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 8m 9s YARN-2915 passed +1 compile 0m 23s YARN-2915 passed +1 checkstyle 0m 15s YARN-2915 passed +1 mvnsite 0m 29s YARN-2915 passed +1 mvneclipse 0m 15s YARN-2915 passed +1 findbugs 0m 59s YARN-2915 passed +1 javadoc 0m 18s YARN-2915 passed +1 mvninstall 0m 25s the patch passed +1 compile 0m 23s the patch passed +1 cc 0m 23s the patch passed +1 javac 0m 23s the patch passed -1 checkstyle 0m 11s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common: The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) +1 mvnsite 0m 29s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 0s the patch passed +1 javadoc 0m 19s the patch passed +1 unit 0m 38s hadoop-yarn-server-common in the patch passed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 15m 45s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822884/YARN-5407-YARN-2915.v2.patch JIRA Issue YARN-5407 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux 07cdcfbc67a7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision YARN-2915 / dbaebf8 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12702/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12702/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common Console output https://builds.apache.org/job/PreCommit-YARN-Build/12702/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          subru Subru Krishnan added a comment -

          +1 on the latest patch, I committed to branch YARN-2915.

          Thanks Ellen Hui for working on this & Jian He for reviewing it.

          Show
          subru Subru Krishnan added a comment - +1 on the latest patch, I committed to branch YARN-2915 . Thanks Ellen Hui for working on this & Jian He for reviewing it.

            People

            • Assignee:
              ellenfkh Ellen Hui
              Reporter:
              subru Subru Krishnan
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development