Details

    • Hadoop Flags:
      Reviewed

      Description

      This JIRA tracks APIs for the policies that will guide the Router and AMRMProxy decisions on where to fwd the jobs submission/query requests as well as ResourceRequests.

      1. YARN-5323.01.patch
        27 kB
        Carlo Curino
      2. YARN-5323.02.patch
        31 kB
        Carlo Curino
      3. YARN-5323.03.patch
        31 kB
        Carlo Curino
      4. YARN-5323.04.patch
        31 kB
        Carlo Curino
      5. YARN-5323-YARN-2915.05.patch
        31 kB
        Carlo Curino
      6. YARN-5323-YARN-2915.06.patch
        37 kB
        Carlo Curino
      7. YARN-5323-YARN-2915.07.patch
        38 kB
        Carlo Curino
      8. YARN-5323-YARN-2915.08.patch
        51 kB
        Carlo Curino
      9. YARN-5323-YARN-2915.09.patch
        49 kB
        Carlo Curino
      10. YARN-5323-YARN-2915.10.patch
        46 kB
        Carlo Curino
      11. YARN-5323-YARN-2915.11.patch
        46 kB
        Carlo Curino

        Issue Links

          Activity

          Hide
          curino Carlo Curino added a comment -

          This initial patch presents the APIs for policies (implementations in YARN-5324, YARN-5325). The key philosophy of our design is to:

          1. have a very opaque format in the federation store, so that we can evolve policy freely (likely to happen as we experience with this). To assist with this though, we foresee each FederationPolicy to implement the FederationPolicyWriter interface that allows us to control the entire lifecycle of a policy. The envisioned usage is: 1) a FederationPolicy is instantiated as part of the admin tools, and configure by setting all sort of configs in it (which might change drastically as policies get more sophisticated), 2) the admin tools invoke serialize() and get a FederationPolicyConfiguration out of it that is stored in the policy store, 3) on the router/amrmproxy side this is fetched, a FederationPolicy object is instantiated, and via the FederationPolicyConfigurator apis the RouterPolicy and AMRMProxyFederationPolicy are instantiated and initialized.
          2. Note that we actively want to bind together the choice of AMRMProxyFederationPolicy and RouterPolicy, which must be coordinated to operate correctly. As such the FederationPolicyConfigurator is a way to raise the bar for a misconfiguration (one needs to write a class that mistakenly combines to policies, instead of just mistyping a conf). This also allows to do consistency checks among the configurations of the two policies.
          Show
          curino Carlo Curino added a comment - This initial patch presents the APIs for policies (implementations in YARN-5324 , YARN-5325 ). The key philosophy of our design is to: have a very opaque format in the federation store, so that we can evolve policy freely (likely to happen as we experience with this). To assist with this though, we foresee each FederationPolicy to implement the FederationPolicyWriter interface that allows us to control the entire lifecycle of a policy. The envisioned usage is: 1) a FederationPolicy is instantiated as part of the admin tools, and configure by setting all sort of configs in it (which might change drastically as policies get more sophisticated), 2) the admin tools invoke serialize() and get a FederationPolicyConfiguration out of it that is stored in the policy store, 3) on the router/amrmproxy side this is fetched, a FederationPolicy object is instantiated, and via the FederationPolicyConfigurator apis the RouterPolicy and AMRMProxyFederationPolicy are instantiated and initialized. Note that we actively want to bind together the choice of AMRMProxyFederationPolicy and RouterPolicy , which must be coordinated to operate correctly. As such the FederationPolicyConfigurator is a way to raise the bar for a misconfiguration (one needs to write a class that mistakenly combines to policies, instead of just mistyping a conf). This also allows to do consistency checks among the configurations of the two policies.
          Hide
          curino Carlo Curino added a comment -

          Adding missing exceptions classes.

          Show
          curino Carlo Curino added a comment - Adding missing exceptions classes.
          Hide
          curino Carlo Curino added a comment -

          Rebased after commit of YARN-5390

          Show
          curino Carlo Curino added a comment - Rebased after commit of YARN-5390
          Hide
          curino Carlo Curino added a comment -

          Wrong version, reuploaded

          Show
          curino Carlo Curino added a comment - Wrong version, reuploaded
          Hide
          curino Carlo Curino added a comment -

          (renaming to target branch for Jenkins)

          Show
          curino Carlo Curino added a comment - (renaming to target branch for Jenkins)
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 1s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 7m 40s trunk passed
          +1 compile 0m 18s trunk passed
          +1 checkstyle 0m 13s trunk passed
          +1 mvnsite 0m 22s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 0m 38s trunk passed
          +1 javadoc 0m 15s trunk passed
          -1 mvninstall 0m 14s hadoop-yarn-server-common in the patch failed.
          -1 compile 0m 13s hadoop-yarn-server-common in the patch failed.
          -1 javac 0m 13s hadoop-yarn-server-common in the patch failed.
          -1 checkstyle 0m 10s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common: The patch generated 18 new + 0 unchanged - 0 fixed = 18 total (was 0)
          -1 mvnsite 0m 14s hadoop-yarn-server-common in the patch failed.
          +1 mvneclipse 0m 9s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          -1 findbugs 0m 13s hadoop-yarn-server-common in the patch failed.
          -1 javadoc 0m 12s hadoop-yarn-server-common in the patch failed.
          -1 unit 0m 13s hadoop-yarn-server-common in the patch failed.
          +1 asflicense 0m 16s The patch does not generate ASF License warnings.
          12m 19s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12823087/YARN-5323.04.patch
          JIRA Issue YARN-5323
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux b3ced6eabeec 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / e83be44
          Default Java 1.8.0_101
          findbugs v3.0.0
          mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/12732/artifact/patchprocess/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt
          compile https://builds.apache.org/job/PreCommit-YARN-Build/12732/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt
          javac https://builds.apache.org/job/PreCommit-YARN-Build/12732/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12732/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt
          mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/12732/artifact/patchprocess/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/12732/artifact/patchprocess/patch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt
          javadoc https://builds.apache.org/job/PreCommit-YARN-Build/12732/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/12732/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12732/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/12732/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 1s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 40s trunk passed +1 compile 0m 18s trunk passed +1 checkstyle 0m 13s trunk passed +1 mvnsite 0m 22s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 38s trunk passed +1 javadoc 0m 15s trunk passed -1 mvninstall 0m 14s hadoop-yarn-server-common in the patch failed. -1 compile 0m 13s hadoop-yarn-server-common in the patch failed. -1 javac 0m 13s hadoop-yarn-server-common in the patch failed. -1 checkstyle 0m 10s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common: The patch generated 18 new + 0 unchanged - 0 fixed = 18 total (was 0) -1 mvnsite 0m 14s hadoop-yarn-server-common in the patch failed. +1 mvneclipse 0m 9s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. -1 findbugs 0m 13s hadoop-yarn-server-common in the patch failed. -1 javadoc 0m 12s hadoop-yarn-server-common in the patch failed. -1 unit 0m 13s hadoop-yarn-server-common in the patch failed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 12m 19s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12823087/YARN-5323.04.patch JIRA Issue YARN-5323 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux b3ced6eabeec 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e83be44 Default Java 1.8.0_101 findbugs v3.0.0 mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/12732/artifact/patchprocess/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt compile https://builds.apache.org/job/PreCommit-YARN-Build/12732/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt javac https://builds.apache.org/job/PreCommit-YARN-Build/12732/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12732/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/12732/artifact/patchprocess/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/12732/artifact/patchprocess/patch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt javadoc https://builds.apache.org/job/PreCommit-YARN-Build/12732/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/12732/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12732/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common Console output https://builds.apache.org/job/PreCommit-YARN-Build/12732/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 10s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 10m 23s YARN-2915 passed
          +1 compile 0m 19s YARN-2915 passed
          +1 checkstyle 0m 15s YARN-2915 passed
          +1 mvnsite 0m 25s YARN-2915 passed
          +1 mvneclipse 0m 16s YARN-2915 passed
          +1 findbugs 0m 46s YARN-2915 passed
          +1 javadoc 0m 17s YARN-2915 passed
          +1 mvninstall 0m 18s the patch passed
          +1 compile 0m 17s the patch passed
          +1 javac 0m 17s the patch passed
          -1 checkstyle 0m 10s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common: The patch generated 18 new + 0 unchanged - 0 fixed = 18 total (was 0)
          +1 mvnsite 0m 20s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 46s the patch passed
          -1 javadoc 0m 13s hadoop-yarn-server-common in the patch failed.
          +1 unit 0m 29s hadoop-yarn-server-common in the patch passed.
          +1 asflicense 0m 19s The patch does not generate ASF License warnings.
          16m 33s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12823156/YARN-5323-YARN-2915.05.patch
          JIRA Issue YARN-5323
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux bd0b8459b0bc 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision YARN-2915 / b689f55
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12736/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt
          javadoc https://builds.apache.org/job/PreCommit-YARN-Build/12736/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12736/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/12736/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 10m 23s YARN-2915 passed +1 compile 0m 19s YARN-2915 passed +1 checkstyle 0m 15s YARN-2915 passed +1 mvnsite 0m 25s YARN-2915 passed +1 mvneclipse 0m 16s YARN-2915 passed +1 findbugs 0m 46s YARN-2915 passed +1 javadoc 0m 17s YARN-2915 passed +1 mvninstall 0m 18s the patch passed +1 compile 0m 17s the patch passed +1 javac 0m 17s the patch passed -1 checkstyle 0m 10s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common: The patch generated 18 new + 0 unchanged - 0 fixed = 18 total (was 0) +1 mvnsite 0m 20s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 46s the patch passed -1 javadoc 0m 13s hadoop-yarn-server-common in the patch failed. +1 unit 0m 29s hadoop-yarn-server-common in the patch passed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 16m 33s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12823156/YARN-5323-YARN-2915.05.patch JIRA Issue YARN-5323 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux bd0b8459b0bc 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision YARN-2915 / b689f55 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12736/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt javadoc https://builds.apache.org/job/PreCommit-YARN-Build/12736/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12736/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common Console output https://builds.apache.org/job/PreCommit-YARN-Build/12736/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Thanks Carlo Curino,

          Some questions / comments:

          1) Who will consume AMRMProxyFederationPolicy and RouterFederationPolicy? I'm not be able to figure it out from the patch, I guess AMRMProxyService will use them, is it correct?
          2) It looks like FederationPolicy is more like a FederationConfigurator/Creator, is it better to remove the FederationPolicy interface? We can either make FederationPolicyConfigurator extends FederationPolicyWriter or let implementation decide which to implement (such as SimpleFederationPolicyConfiguratorImpl implements FederationPolicyWriter, FederationPolicyConfigurator).
          3) Is it better to rename FederationPolicyContext to FederationPolicyInitializationContext?
          4) The name "router" is a little hard to be understood, now it only determines application's home sub cluster while submitting the app, do you plan to add more functionalities to the module? Do you think is it better to rename it to be "AMRMProxyHomeClusterPolicy", (and similarly we may rename AMRMProxyFederationPolicy to AMRMProxyResourceRequestPolicy).

          Show
          leftnoteasy Wangda Tan added a comment - Thanks Carlo Curino , Some questions / comments: 1) Who will consume AMRMProxyFederationPolicy and RouterFederationPolicy? I'm not be able to figure it out from the patch, I guess AMRMProxyService will use them, is it correct? 2) It looks like FederationPolicy is more like a FederationConfigurator/Creator, is it better to remove the FederationPolicy interface? We can either make FederationPolicyConfigurator extends FederationPolicyWriter or let implementation decide which to implement (such as SimpleFederationPolicyConfiguratorImpl implements FederationPolicyWriter, FederationPolicyConfigurator ). 3) Is it better to rename FederationPolicyContext to FederationPolicyInitializationContext? 4) The name "router" is a little hard to be understood, now it only determines application's home sub cluster while submitting the app, do you plan to add more functionalities to the module? Do you think is it better to rename it to be "AMRMProxyHomeClusterPolicy", (and similarly we may rename AMRMProxyFederationPolicy to AMRMProxyResourceRequestPolicy).
          Hide
          curino Carlo Curino added a comment - - edited

          Tan, Wangda thanks for reviewing.

          Answering your questions:

          1. The AMRMProxyFederationPolicy is used in the AMRMProxyService to determine where to send the ResourceRequests, while the RouterFederationPolicy is used by the router to determine where to send the AM. This patch is a bit of a "leaf" patch, other patches will use this code, so it make sense you don't get much context from this patch alone.
          2. We could remove it, but the key intuition we have is that the two AMRMProxyPolicy and RouterPolicy, while completely independent (run on different nodes and do different things), need to work in collaboration with each other to achieve a consistent system behavior. We are thus trying to use the code structure (e.g., FederationPolicy) as a way to tie them together. This should help administrators to avoid some of the obvious mistakes (e.g., combining a load-balancing RouterPolicy with a fault-tolerance focused AMRMProxyPolicy)---if you want you can, but have to implement a class and configure the system to use it, which should make you think harder about what you are doing. Also we are using this writer + getter api as a way to handle the entire lifecycle of ser/deser of the policy configuration through the store. By construction we want the configuration to be rather opaque (expandibility), but we need to keep the lifecycle management sane, this is our attempt.
          3. Works for me, though I don't think is the common way for other "*Context" objects around YARN.
          4. The Router routes more than the APPs, all client-to-rm protocols go through there... Most of the decisions of where to route are not policy based, as they are deterministic (e.g., if you send a kill should go where the app was running). I think the policy name should say Router to indicate that is used by the router. Also in longer term I can see more policy behaviors to be added to the router, as we become more sophisticated and operations that are fully manual today (capacity allocation to subclusters) get automated.

          I hope this helps to clarify your concerns.

          Show
          curino Carlo Curino added a comment - - edited Tan, Wangda thanks for reviewing. Answering your questions: The AMRMProxyFederationPolicy is used in the AMRMProxyService to determine where to send the ResourceRequests, while the RouterFederationPolicy is used by the router to determine where to send the AM. This patch is a bit of a "leaf" patch, other patches will use this code, so it make sense you don't get much context from this patch alone. We could remove it, but the key intuition we have is that the two AMRMProxyPolicy and RouterPolicy, while completely independent (run on different nodes and do different things), need to work in collaboration with each other to achieve a consistent system behavior. We are thus trying to use the code structure (e.g., FederationPolicy) as a way to tie them together. This should help administrators to avoid some of the obvious mistakes (e.g., combining a load-balancing RouterPolicy with a fault-tolerance focused AMRMProxyPolicy)---if you want you can, but have to implement a class and configure the system to use it, which should make you think harder about what you are doing. Also we are using this writer + getter api as a way to handle the entire lifecycle of ser/deser of the policy configuration through the store. By construction we want the configuration to be rather opaque (expandibility), but we need to keep the lifecycle management sane, this is our attempt. Works for me, though I don't think is the common way for other "*Context" objects around YARN. The Router routes more than the APPs, all client-to-rm protocols go through there... Most of the decisions of where to route are not policy based, as they are deterministic (e.g., if you send a kill should go where the app was running). I think the policy name should say Router to indicate that is used by the router. Also in longer term I can see more policy behaviors to be added to the router, as we become more sophisticated and operations that are fully manual today (capacity allocation to subclusters) get automated. I hope this helps to clarify your concerns.
          Hide
          curino Carlo Curino added a comment -

          Regarding (2) I think looking at YARN-5391 might help to get a sense. The implementations are used to "tie" the AMRM and Router policies together.

          Show
          curino Carlo Curino added a comment - Regarding (2) I think looking at YARN-5391 might help to get a sense. The implementations are used to "tie" the AMRM and Router policies together.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Thanks Carlo Curino for explanations, now I can better understand background.

          For 2). It is make sense to have a unified configurator to instance different type of policies. But I still think the name is a little bit confusing, do you think is it better to rename it to something like FederationPoliciesConfigurator?

          For 3). There're some classes such as ApplicationInitializationContext / ContainerInitializationContext.

          Show
          leftnoteasy Wangda Tan added a comment - Thanks Carlo Curino for explanations, now I can better understand background. For 2). It is make sense to have a unified configurator to instance different type of policies. But I still think the name is a little bit confusing, do you think is it better to rename it to something like FederationPoliciesConfigurator? For 3). There're some classes such as ApplicationInitializationContext / ContainerInitializationContext.
          Hide
          curino Carlo Curino added a comment -

          No problem renaming 2, 3 as you suggest. Other than this the patch looks ok?

          Show
          curino Carlo Curino added a comment - No problem renaming 2, 3 as you suggest. Other than this the patch looks ok?
          Hide
          leftnoteasy Wangda Tan added a comment -

          Yeah other than that, patch looks fine

          Show
          leftnoteasy Wangda Tan added a comment - Yeah other than that, patch looks fine
          Hide
          curino Carlo Curino added a comment -

          Tan, Wangda, I addressed 2 and 3 above... I use FederationPolicyManager for (2) as FederationPoliciesConfigurator clashes
          a bit with FederationPolicyConfigurator. I also tried to address the checkstyle/javadoc (let's see what Jenkins thinks).

          Subru Krishnan and Giovanni Matteo Fumarola if you have time, take a look as well. You have lots of context and are working on some of
          the code that will use it. (It is ok to also have "improvement" type of comments for the V2 of federation, we can track them in
          subjiras of YARN-5597 as appropriate).

          Thanks,
          Carlo

          Show
          curino Carlo Curino added a comment - Tan, Wangda , I addressed 2 and 3 above... I use FederationPolicyManager for (2) as FederationPoliciesConfigurator clashes a bit with FederationPolicyConfigurator. I also tried to address the checkstyle/javadoc (let's see what Jenkins thinks). Subru Krishnan and Giovanni Matteo Fumarola if you have time, take a look as well. You have lots of context and are working on some of the code that will use it. (It is ok to also have "improvement" type of comments for the V2 of federation, we can track them in subjiras of YARN-5597 as appropriate). Thanks, Carlo
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 7m 56s YARN-2915 passed
          +1 compile 0m 23s YARN-2915 passed
          +1 checkstyle 0m 14s YARN-2915 passed
          +1 mvnsite 0m 25s YARN-2915 passed
          +1 mvneclipse 0m 14s YARN-2915 passed
          +1 findbugs 0m 48s YARN-2915 passed
          +1 javadoc 0m 17s YARN-2915 passed
          +1 mvninstall 0m 21s the patch passed
          +1 compile 0m 21s the patch passed
          +1 javac 0m 21s the patch passed
          -1 checkstyle 0m 12s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common: The patch generated 9 new + 0 unchanged - 0 fixed = 9 total (was 0)
          +1 mvnsite 0m 23s the patch passed
          +1 mvneclipse 0m 12s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 55s the patch passed
          -1 javadoc 0m 15s hadoop-yarn-server-common in the patch failed.
          +1 unit 0m 34s hadoop-yarn-server-common in the patch passed.
          +1 asflicense 0m 16s The patch does not generate ASF License warnings.
          14m 47s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12826263/YARN-5323-YARN-2915.06.patch
          JIRA Issue YARN-5323
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 7277211f4d91 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision YARN-2915 / c77269d
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12956/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt
          javadoc https://builds.apache.org/job/PreCommit-YARN-Build/12956/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12956/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/12956/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 56s YARN-2915 passed +1 compile 0m 23s YARN-2915 passed +1 checkstyle 0m 14s YARN-2915 passed +1 mvnsite 0m 25s YARN-2915 passed +1 mvneclipse 0m 14s YARN-2915 passed +1 findbugs 0m 48s YARN-2915 passed +1 javadoc 0m 17s YARN-2915 passed +1 mvninstall 0m 21s the patch passed +1 compile 0m 21s the patch passed +1 javac 0m 21s the patch passed -1 checkstyle 0m 12s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common: The patch generated 9 new + 0 unchanged - 0 fixed = 9 total (was 0) +1 mvnsite 0m 23s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 55s the patch passed -1 javadoc 0m 15s hadoop-yarn-server-common in the patch failed. +1 unit 0m 34s hadoop-yarn-server-common in the patch passed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 14m 47s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12826263/YARN-5323-YARN-2915.06.patch JIRA Issue YARN-5323 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 7277211f4d91 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision YARN-2915 / c77269d Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12956/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt javadoc https://builds.apache.org/job/PreCommit-YARN-Build/12956/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12956/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common Console output https://builds.apache.org/job/PreCommit-YARN-Build/12956/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          giovanni.fumarola Giovanni Matteo Fumarola added a comment -

          LGTM.

          Minor fix:
          1) AMRMProxyFederationPolicy -> FederationAMRMProxyPolicy;
          2) RouterFederationPolicy -> FederationRouterPolicy;

          Show
          giovanni.fumarola Giovanni Matteo Fumarola added a comment - LGTM. Minor fix: 1) AMRMProxyFederationPolicy -> FederationAMRMProxyPolicy; 2) RouterFederationPolicy -> FederationRouterPolicy;
          Hide
          leftnoteasy Wangda Tan added a comment -

          Thanks Carlo Curino,

          Thanks for update, patch looks good in my perspective. I think it's better to let Subru Krishnan commit it since he has more contexts.

          And for the implementation of the policies YARN-5324/YARN-5325, I may not have bandwidth in short term. Subru Krishnan, could you take care of patch review for the two implementation patches?

          Thanks,

          Show
          leftnoteasy Wangda Tan added a comment - Thanks Carlo Curino , Thanks for update, patch looks good in my perspective. I think it's better to let Subru Krishnan commit it since he has more contexts. And for the implementation of the policies YARN-5324 / YARN-5325 , I may not have bandwidth in short term. Subru Krishnan , could you take care of patch review for the two implementation patches? Thanks,
          Hide
          subru Subru Krishnan added a comment -

          Sure Wangda Tan, I'll take a look. Thanks.

          Show
          subru Subru Krishnan added a comment - Sure Wangda Tan , I'll take a look. Thanks.
          Hide
          subru Subru Krishnan added a comment -

          Thanks Carlo Curino for working on this. The latest patch mostly LGTM, have a few minor comments:

          • I feel it would be better to have FederationStateStoreFacade in FederationPolicyInitializationContext and use that instead of passing active sub-cluster map in every invocation, that too of both Router/AMRMProxyFederationPolicy.
          • There are few public methods missing Javadocs like getters/setters in FederationPolicyInitializationContext.
          • IMO few of the open Yetus checkstyle/javadoc warnings are fixable.
          Show
          subru Subru Krishnan added a comment - Thanks Carlo Curino for working on this. The latest patch mostly LGTM, have a few minor comments: I feel it would be better to have FederationStateStoreFacade in FederationPolicyInitializationContext and use that instead of passing active sub-cluster map in every invocation, that too of both Router/AMRMProxyFederationPolicy . There are few public methods missing Javadocs like getters/setters in FederationPolicyInitializationContext . IMO few of the open Yetus checkstyle/javadoc warnings are fixable.
          Hide
          curino Carlo Curino added a comment -

          Giovanni Matteo Fumarola and Subru Krishnan thanks for your comments.

          I addressed all of them expect Subru Krishnan's FederationStateStoreFacade one.
          The rationale for leaving the API as is, is that it makes it evident to the implementor that
          the set of active sub-cluster could be changing over time, and should be handle with care.
          This way the code is (not completely, but close to) functional, which I like for the policies.

          Show
          curino Carlo Curino added a comment - Giovanni Matteo Fumarola and Subru Krishnan thanks for your comments. I addressed all of them expect Subru Krishnan 's FederationStateStoreFacade one. The rationale for leaving the API as is, is that it makes it evident to the implementor that the set of active sub-cluster could be changing over time, and should be handle with care. This way the code is (not completely, but close to) functional, which I like for the policies.
          Hide
          curino Carlo Curino added a comment -

          Actually, Subru Krishnan convinced me about the Facade, and helped fixing the patch attached now (.07).

          Show
          curino Carlo Curino added a comment - Actually, Subru Krishnan convinced me about the Facade, and helped fixing the patch attached now (.07).
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 7m 31s YARN-2915 passed
          +1 compile 0m 22s YARN-2915 passed
          +1 checkstyle 0m 14s YARN-2915 passed
          +1 mvnsite 0m 24s YARN-2915 passed
          +1 mvneclipse 0m 12s YARN-2915 passed
          +1 findbugs 0m 47s YARN-2915 passed
          +1 javadoc 0m 18s YARN-2915 passed
          +1 mvninstall 0m 23s the patch passed
          +1 compile 0m 20s the patch passed
          +1 javac 0m 20s the patch passed
          -1 checkstyle 0m 11s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common: The patch generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0)
          +1 mvnsite 0m 23s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 56s the patch passed
          +1 javadoc 0m 15s the patch passed
          +1 unit 0m 33s hadoop-yarn-server-common in the patch passed.
          +1 asflicense 0m 16s The patch does not generate ASF License warnings.
          14m 15s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12826525/YARN-5323-YARN-2915.07.patch
          JIRA Issue YARN-5323
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 6b59a1a54fe8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision YARN-2915 / c77269d
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12979/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12979/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/12979/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 31s YARN-2915 passed +1 compile 0m 22s YARN-2915 passed +1 checkstyle 0m 14s YARN-2915 passed +1 mvnsite 0m 24s YARN-2915 passed +1 mvneclipse 0m 12s YARN-2915 passed +1 findbugs 0m 47s YARN-2915 passed +1 javadoc 0m 18s YARN-2915 passed +1 mvninstall 0m 23s the patch passed +1 compile 0m 20s the patch passed +1 javac 0m 20s the patch passed -1 checkstyle 0m 11s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common: The patch generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0) +1 mvnsite 0m 23s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 56s the patch passed +1 javadoc 0m 15s the patch passed +1 unit 0m 33s hadoop-yarn-server-common in the patch passed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 14m 15s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12826525/YARN-5323-YARN-2915.07.patch JIRA Issue YARN-5323 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 6b59a1a54fe8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision YARN-2915 / c77269d Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12979/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12979/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common Console output https://builds.apache.org/job/PreCommit-YARN-Build/12979/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          giovanni.fumarola Giovanni Matteo Fumarola added a comment -

          Thanks Carlo Curino for the patch.
          Minor fixes:
          1) In FederationPolicyInitializationContext, move private FederationStateStoreFacade federationStateStoreFacade; next to the other variables.
          2) In FederationPolicyInitializationContext, add constructor.
          3) Add test for FederationPolicyInitializationContextValidator (same as TestFederationStateStoreInputValidator). In this way we will fix also test4tests.

          Show
          giovanni.fumarola Giovanni Matteo Fumarola added a comment - Thanks Carlo Curino for the patch. Minor fixes: 1) In FederationPolicyInitializationContext, move private FederationStateStoreFacade federationStateStoreFacade; next to the other variables. 2) In FederationPolicyInitializationContext, add constructor. 3) Add test for FederationPolicyInitializationContextValidator (same as TestFederationStateStoreInputValidator). In this way we will fix also test4tests.
          Hide
          curino Carlo Curino added a comment -

          Giovanni Matteo Fumarola thanks for the comments, I addressed them in the latest (.08) patch.

          Show
          curino Carlo Curino added a comment - Giovanni Matteo Fumarola thanks for the comments, I addressed them in the latest (.08) patch.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 7m 14s YARN-2915 passed
          +1 compile 0m 21s YARN-2915 passed
          +1 checkstyle 0m 14s YARN-2915 passed
          +1 mvnsite 0m 24s YARN-2915 passed
          +1 mvneclipse 0m 13s YARN-2915 passed
          +1 findbugs 0m 42s YARN-2915 passed
          +1 javadoc 0m 15s YARN-2915 passed
          -1 mvninstall 0m 17s hadoop-yarn-server-common in the patch failed.
          -1 compile 0m 18s hadoop-yarn-server-common in the patch failed.
          -1 javac 0m 18s hadoop-yarn-server-common in the patch failed.
          -1 checkstyle 0m 10s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common: The patch generated 7 new + 0 unchanged - 0 fixed = 7 total (was 0)
          -1 mvnsite 0m 18s hadoop-yarn-server-common in the patch failed.
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          -1 findbugs 0m 11s hadoop-yarn-server-common in the patch failed.
          +1 javadoc 0m 13s the patch passed
          -1 unit 0m 17s hadoop-yarn-server-common in the patch failed.
          -1 asflicense 0m 15s The patch generated 1 ASF License warnings.
          12m 25s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12826726/YARN-5323-YARN-2915.08.patch
          JIRA Issue YARN-5323
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux a512bd6960b3 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision YARN-2915 / bd906b2
          Default Java 1.8.0_101
          findbugs v3.0.0
          mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/13002/artifact/patchprocess/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt
          compile https://builds.apache.org/job/PreCommit-YARN-Build/13002/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt
          javac https://builds.apache.org/job/PreCommit-YARN-Build/13002/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13002/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt
          mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/13002/artifact/patchprocess/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/13002/artifact/patchprocess/patch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/13002/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13002/testReport/
          asflicense https://builds.apache.org/job/PreCommit-YARN-Build/13002/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/13002/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 7m 14s YARN-2915 passed +1 compile 0m 21s YARN-2915 passed +1 checkstyle 0m 14s YARN-2915 passed +1 mvnsite 0m 24s YARN-2915 passed +1 mvneclipse 0m 13s YARN-2915 passed +1 findbugs 0m 42s YARN-2915 passed +1 javadoc 0m 15s YARN-2915 passed -1 mvninstall 0m 17s hadoop-yarn-server-common in the patch failed. -1 compile 0m 18s hadoop-yarn-server-common in the patch failed. -1 javac 0m 18s hadoop-yarn-server-common in the patch failed. -1 checkstyle 0m 10s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common: The patch generated 7 new + 0 unchanged - 0 fixed = 7 total (was 0) -1 mvnsite 0m 18s hadoop-yarn-server-common in the patch failed. +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. -1 findbugs 0m 11s hadoop-yarn-server-common in the patch failed. +1 javadoc 0m 13s the patch passed -1 unit 0m 17s hadoop-yarn-server-common in the patch failed. -1 asflicense 0m 15s The patch generated 1 ASF License warnings. 12m 25s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12826726/YARN-5323-YARN-2915.08.patch JIRA Issue YARN-5323 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux a512bd6960b3 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision YARN-2915 / bd906b2 Default Java 1.8.0_101 findbugs v3.0.0 mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/13002/artifact/patchprocess/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt compile https://builds.apache.org/job/PreCommit-YARN-Build/13002/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt javac https://builds.apache.org/job/PreCommit-YARN-Build/13002/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13002/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/13002/artifact/patchprocess/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/13002/artifact/patchprocess/patch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13002/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13002/testReport/ asflicense https://builds.apache.org/job/PreCommit-YARN-Build/13002/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common Console output https://builds.apache.org/job/PreCommit-YARN-Build/13002/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          curino Carlo Curino added a comment -

          The last version included an extra class with broken dependencies (needs YARN-5324, YARN-5325). I resolved this, and polished few more checkstyles.

          Show
          curino Carlo Curino added a comment - The last version included an extra class with broken dependencies (needs YARN-5324 , YARN-5325 ). I resolved this, and polished few more checkstyles.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 8m 49s YARN-2915 passed
          +1 compile 0m 23s YARN-2915 passed
          +1 checkstyle 0m 15s YARN-2915 passed
          +1 mvnsite 0m 29s YARN-2915 passed
          +1 mvneclipse 0m 14s YARN-2915 passed
          +1 findbugs 0m 51s YARN-2915 passed
          +1 javadoc 0m 19s YARN-2915 passed
          +1 mvninstall 0m 23s the patch passed
          +1 compile 0m 20s the patch passed
          +1 javac 0m 20s the patch passed
          -1 checkstyle 0m 12s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common: The patch generated 6 new + 0 unchanged - 0 fixed = 6 total (was 0)
          +1 mvnsite 0m 25s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 2s the patch passed
          +1 javadoc 0m 14s the patch passed
          +1 unit 0m 35s hadoop-yarn-server-common in the patch passed.
          +1 asflicense 0m 17s The patch does not generate ASF License warnings.
          15m 59s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12827259/YARN-5323-YARN-2915.09.patch
          JIRA Issue YARN-5323
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 069d87fd1c25 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision YARN-2915 / f2985a3
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13022/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13022/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/13022/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 8m 49s YARN-2915 passed +1 compile 0m 23s YARN-2915 passed +1 checkstyle 0m 15s YARN-2915 passed +1 mvnsite 0m 29s YARN-2915 passed +1 mvneclipse 0m 14s YARN-2915 passed +1 findbugs 0m 51s YARN-2915 passed +1 javadoc 0m 19s YARN-2915 passed +1 mvninstall 0m 23s the patch passed +1 compile 0m 20s the patch passed +1 javac 0m 20s the patch passed -1 checkstyle 0m 12s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common: The patch generated 6 new + 0 unchanged - 0 fixed = 6 total (was 0) +1 mvnsite 0m 25s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 2s the patch passed +1 javadoc 0m 14s the patch passed +1 unit 0m 35s hadoop-yarn-server-common in the patch passed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 15m 59s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12827259/YARN-5323-YARN-2915.09.patch JIRA Issue YARN-5323 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 069d87fd1c25 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision YARN-2915 / f2985a3 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13022/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13022/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common Console output https://builds.apache.org/job/PreCommit-YARN-Build/13022/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          curino Carlo Curino added a comment -

          More checkstyle fixed, and removed FederationPolicyManager, which was somewhat redundant.

          Show
          curino Carlo Curino added a comment - More checkstyle fixed, and removed FederationPolicyManager, which was somewhat redundant.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 7m 9s YARN-2915 passed
          +1 compile 0m 20s YARN-2915 passed
          +1 checkstyle 0m 14s YARN-2915 passed
          +1 mvnsite 0m 24s YARN-2915 passed
          +1 mvneclipse 0m 14s YARN-2915 passed
          +1 findbugs 0m 42s YARN-2915 passed
          +1 javadoc 0m 17s YARN-2915 passed
          +1 mvninstall 0m 19s the patch passed
          +1 compile 0m 18s the patch passed
          +1 javac 0m 18s the patch passed
          +1 checkstyle 0m 11s the patch passed
          +1 mvnsite 0m 20s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 46s the patch passed
          -1 javadoc 0m 17s hadoop-yarn-server-common in the patch failed.
          +1 unit 0m 33s hadoop-yarn-server-common in the patch passed.
          +1 asflicense 0m 15s The patch does not generate ASF License warnings.
          13m 33s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12827274/YARN-5323-YARN-2915.10.patch
          JIRA Issue YARN-5323
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 9aa8c6135e4d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision YARN-2915 / f2985a3
          Default Java 1.8.0_101
          findbugs v3.0.0
          javadoc https://builds.apache.org/job/PreCommit-YARN-Build/13025/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13025/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/13025/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 7m 9s YARN-2915 passed +1 compile 0m 20s YARN-2915 passed +1 checkstyle 0m 14s YARN-2915 passed +1 mvnsite 0m 24s YARN-2915 passed +1 mvneclipse 0m 14s YARN-2915 passed +1 findbugs 0m 42s YARN-2915 passed +1 javadoc 0m 17s YARN-2915 passed +1 mvninstall 0m 19s the patch passed +1 compile 0m 18s the patch passed +1 javac 0m 18s the patch passed +1 checkstyle 0m 11s the patch passed +1 mvnsite 0m 20s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 46s the patch passed -1 javadoc 0m 17s hadoop-yarn-server-common in the patch failed. +1 unit 0m 33s hadoop-yarn-server-common in the patch passed. +1 asflicense 0m 15s The patch does not generate ASF License warnings. 13m 33s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12827274/YARN-5323-YARN-2915.10.patch JIRA Issue YARN-5323 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 9aa8c6135e4d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision YARN-2915 / f2985a3 Default Java 1.8.0_101 findbugs v3.0.0 javadoc https://builds.apache.org/job/PreCommit-YARN-Build/13025/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13025/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common Console output https://builds.apache.org/job/PreCommit-YARN-Build/13025/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 20s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 7m 8s YARN-2915 passed
          +1 compile 0m 20s YARN-2915 passed
          +1 checkstyle 0m 12s YARN-2915 passed
          +1 mvnsite 0m 23s YARN-2915 passed
          +1 mvneclipse 0m 12s YARN-2915 passed
          +1 findbugs 0m 47s YARN-2915 passed
          +1 javadoc 0m 16s YARN-2915 passed
          +1 mvninstall 0m 19s the patch passed
          +1 compile 0m 17s the patch passed
          +1 javac 0m 17s the patch passed
          +1 checkstyle 0m 10s the patch passed
          +1 mvnsite 0m 21s the patch passed
          +1 mvneclipse 0m 9s the patch passed
          +1 whitespace 0m 1s The patch has no whitespace issues.
          +1 findbugs 0m 46s the patch passed
          +1 javadoc 0m 13s the patch passed
          +1 unit 0m 32s hadoop-yarn-server-common in the patch passed.
          +1 asflicense 0m 16s The patch does not generate ASF License warnings.
          13m 25s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12827400/YARN-5323-YARN-2915.11.patch
          JIRA Issue YARN-5323
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 7ec8d32f72dc 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision YARN-2915 / f2985a3
          Default Java 1.8.0_101
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13033/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/13033/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 7m 8s YARN-2915 passed +1 compile 0m 20s YARN-2915 passed +1 checkstyle 0m 12s YARN-2915 passed +1 mvnsite 0m 23s YARN-2915 passed +1 mvneclipse 0m 12s YARN-2915 passed +1 findbugs 0m 47s YARN-2915 passed +1 javadoc 0m 16s YARN-2915 passed +1 mvninstall 0m 19s the patch passed +1 compile 0m 17s the patch passed +1 javac 0m 17s the patch passed +1 checkstyle 0m 10s the patch passed +1 mvnsite 0m 21s the patch passed +1 mvneclipse 0m 9s the patch passed +1 whitespace 0m 1s The patch has no whitespace issues. +1 findbugs 0m 46s the patch passed +1 javadoc 0m 13s the patch passed +1 unit 0m 32s hadoop-yarn-server-common in the patch passed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 13m 25s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12827400/YARN-5323-YARN-2915.11.patch JIRA Issue YARN-5323 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 7ec8d32f72dc 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision YARN-2915 / f2985a3 Default Java 1.8.0_101 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13033/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common Console output https://builds.apache.org/job/PreCommit-YARN-Build/13033/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          giovanni.fumarola Giovanni Matteo Fumarola added a comment - - edited

          +1 (non-binding)
          Thanks Carlo Curino.

          Show
          giovanni.fumarola Giovanni Matteo Fumarola added a comment - - edited +1 (non-binding) Thanks Carlo Curino .
          Hide
          subru Subru Krishnan added a comment -

          The latest patch LGTM too, committing it shortly.

          Show
          subru Subru Krishnan added a comment - The latest patch LGTM too, committing it shortly.
          Hide
          subru Subru Krishnan added a comment -

          Thanks Carlo Curino for working on this and Giovanni Matteo Fumarola for reviewing. I just committed this to branch YARN-2915.

          Show
          subru Subru Krishnan added a comment - Thanks Carlo Curino for working on this and Giovanni Matteo Fumarola for reviewing. I just committed this to branch YARN-2915 .
          Hide
          curino Carlo Curino added a comment -

          Thanks Subru Krishnan for reviews and committing.

          Show
          curino Carlo Curino added a comment - Thanks Subru Krishnan for reviews and committing.

            People

            • Assignee:
              curino Carlo Curino
              Reporter:
              curino Carlo Curino
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development