Details

    • Hadoop Flags:
      Reviewed

      Description

      The federation Policy Store contains information about the capacity allocations made by users, their mapping to sub-clusters and the policies that each of the components (Router, AMRMPRoxy, RMs) should enforce

      1. YARN-3664-YARN-2915-v0.patch
        38 kB
        Subru Krishnan
      2. YARN-3664-YARN-2915-v1.patch
        44 kB
        Subru Krishnan
      3. YARN-3664-YARN-2915-v2.patch
        68 kB
        Subru Krishnan
      4. YARN-3664-YARN-2915-v3.patch
        68 kB
        Subru Krishnan
      5. YARN-3664-YARN-2915-v4.patch
        68 kB
        Subru Krishnan

        Issue Links

          Activity

          Hide
          subru Subru Krishnan added a comment -

          I just committed this to branch YARN-2915. Thanks Varun Vasudev and Wangda Tan for the thoughtful reviews.

          Show
          subru Subru Krishnan added a comment - I just committed this to branch YARN-2915 . Thanks Varun Vasudev and Wangda Tan for the thoughtful reviews.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 2s YARN-2915 passed
          +1 compile 0m 20s YARN-2915 passed
          +1 checkstyle 0m 13s YARN-2915 passed
          +1 mvnsite 0m 23s YARN-2915 passed
          +1 mvneclipse 0m 12s YARN-2915 passed
          +1 findbugs 0m 40s YARN-2915 passed
          +1 javadoc 0m 16s YARN-2915 passed
          +1 mvninstall 0m 18s the patch passed
          +1 compile 0m 17s the patch passed
          +1 cc 0m 17s the patch passed
          +1 javac 0m 17s the patch passed
          +1 checkstyle 0m 10s the patch passed
          +1 mvnsite 0m 20s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 45s the patch passed
          +1 javadoc 0m 13s the patch passed
          +1 unit 0m 28s hadoop-yarn-server-common in the patch passed.
          +1 asflicense 0m 16s The patch does not generate ASF License warnings.
          12m 59s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822356/YARN-3664-YARN-2915-v4.patch
          JIRA Issue YARN-3664
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux fb35e0cec049 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision YARN-2915 / f6c60fb
          Default Java 1.8.0_101
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12659/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/12659/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 2s YARN-2915 passed +1 compile 0m 20s YARN-2915 passed +1 checkstyle 0m 13s YARN-2915 passed +1 mvnsite 0m 23s YARN-2915 passed +1 mvneclipse 0m 12s YARN-2915 passed +1 findbugs 0m 40s YARN-2915 passed +1 javadoc 0m 16s YARN-2915 passed +1 mvninstall 0m 18s the patch passed +1 compile 0m 17s the patch passed +1 cc 0m 17s the patch passed +1 javac 0m 17s the patch passed +1 checkstyle 0m 10s the patch passed +1 mvnsite 0m 20s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 45s the patch passed +1 javadoc 0m 13s the patch passed +1 unit 0m 28s hadoop-yarn-server-common in the patch passed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 12m 59s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822356/YARN-3664-YARN-2915-v4.patch JIRA Issue YARN-3664 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux fb35e0cec049 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision YARN-2915 / f6c60fb Default Java 1.8.0_101 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12659/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common Console output https://builds.apache.org/job/PreCommit-YARN-Build/12659/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          subru Subru Krishnan added a comment -

          Same patch as before but after rebasing with YARN-5307. Trigger Yetus before committing.

          Show
          subru Subru Krishnan added a comment - Same patch as before but after rebasing with YARN-5307 . Trigger Yetus before committing.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Make sense, +1 to latest patch, thanks Subru Krishnan.

          Show
          leftnoteasy Wangda Tan added a comment - Make sense, +1 to latest patch, thanks Subru Krishnan .
          Hide
          subru Subru Krishnan added a comment -

          Thanks Wangda Tan for your feedback.

          I agree that using enums wherever possible is better than String. But in the case of types, they are extendible/pluggable and so we see the number growing. For e.g.: We already have 4-5 as part of YARN-5324/YARN-5325 and ideas for a couple more based on how these work out in testing.

          Show
          subru Subru Krishnan added a comment - Thanks Wangda Tan for your feedback. I agree that using enums wherever possible is better than String. But in the case of types, they are extendible/pluggable and so we see the number growing. For e.g.: We already have 4-5 as part of YARN-5324 / YARN-5325 and ideas for a couple more based on how these work out in testing.
          Hide
          subru Subru Krishnan added a comment -

          Thanks Varun Vasudev for reviewing the patch.

          Yes, I intentionally renamed addReservationResourcesToProto to addSubClusterInfosToProto as it was originally a copy-paste typo.

          I'll take care of it as soon as Wangda Tan signs off.

          Show
          subru Subru Krishnan added a comment - Thanks Varun Vasudev for reviewing the patch. Yes, I intentionally renamed addReservationResourcesToProto to addSubClusterInfosToProto as it was originally a copy-paste typo. I'll take care of it as soon as Wangda Tan signs off.
          Hide
          leftnoteasy Wangda Tan added a comment - - edited

          Subru Krishnan,

          One minor comment:

          Can we change getType of SubClusterPolicyConfiguration from String to enum. Using enum should be much better for readability if the type comes from limited choices. And this could also avoid potential bugs.

          Show
          leftnoteasy Wangda Tan added a comment - - edited Subru Krishnan , One minor comment: Can we change getType of SubClusterPolicyConfiguration from String to enum. Using enum should be much better for readability if the type comes from limited choices. And this could also avoid potential bugs.
          Hide
          vvasudev Varun Vasudev added a comment -

          Thanks for the updated patch Subru Krishnan.

          -      addReservationResourcesToProto();
          +      addSubClusterInfosToProto();
          

          Is this correct? Should both be added to the proto or are you fixing a bug in the earlier version?

          +1 apart from the comment above. Let me know if I need to commit the patch or you'll take care of it.

          Show
          vvasudev Varun Vasudev added a comment - Thanks for the updated patch Subru Krishnan . - addReservationResourcesToProto(); + addSubClusterInfosToProto(); Is this correct? Should both be added to the proto or are you fixing a bug in the earlier version? +1 apart from the comment above. Let me know if I need to commit the patch or you'll take care of it.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 20s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 13m 49s YARN-2915 passed
          +1 compile 0m 22s YARN-2915 passed
          +1 checkstyle 0m 16s YARN-2915 passed
          +1 mvnsite 0m 30s YARN-2915 passed
          +1 mvneclipse 0m 18s YARN-2915 passed
          +1 findbugs 0m 54s YARN-2915 passed
          +1 javadoc 0m 19s YARN-2915 passed
          +1 mvninstall 0m 22s the patch passed
          +1 compile 0m 21s the patch passed
          +1 cc 0m 21s the patch passed
          +1 javac 0m 21s the patch passed
          +1 checkstyle 0m 11s the patch passed
          +1 mvnsite 0m 24s the patch passed
          +1 mvneclipse 0m 12s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 55s the patch passed
          +1 javadoc 0m 14s the patch passed
          +1 unit 0m 30s hadoop-yarn-server-common in the patch passed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          20m 56s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12821736/YARN-3664-YARN-2915-v3.patch
          JIRA Issue YARN-3664
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux d0051a266a11 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision YARN-2915 / 22db8fd
          Default Java 1.8.0_101
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12621/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/12621/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 13m 49s YARN-2915 passed +1 compile 0m 22s YARN-2915 passed +1 checkstyle 0m 16s YARN-2915 passed +1 mvnsite 0m 30s YARN-2915 passed +1 mvneclipse 0m 18s YARN-2915 passed +1 findbugs 0m 54s YARN-2915 passed +1 javadoc 0m 19s YARN-2915 passed +1 mvninstall 0m 22s the patch passed +1 compile 0m 21s the patch passed +1 cc 0m 21s the patch passed +1 javac 0m 21s the patch passed +1 checkstyle 0m 11s the patch passed +1 mvnsite 0m 24s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 55s the patch passed +1 javadoc 0m 14s the patch passed +1 unit 0m 30s hadoop-yarn-server-common in the patch passed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 20m 56s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12821736/YARN-3664-YARN-2915-v3.patch JIRA Issue YARN-3664 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux d0051a266a11 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision YARN-2915 / 22db8fd Default Java 1.8.0_101 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12621/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common Console output https://builds.apache.org/job/PreCommit-YARN-Build/12621/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          subru Subru Krishnan added a comment -

          Fixing Yetus warnings (v3).

          Show
          subru Subru Krishnan added a comment - Fixing Yetus warnings (v3).
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 20s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 16s YARN-2915 passed
          +1 compile 0m 20s YARN-2915 passed
          +1 checkstyle 0m 13s YARN-2915 passed
          +1 mvnsite 0m 23s YARN-2915 passed
          +1 mvneclipse 0m 13s YARN-2915 passed
          +1 findbugs 0m 40s YARN-2915 passed
          +1 javadoc 0m 16s YARN-2915 passed
          +1 mvninstall 0m 18s the patch passed
          +1 compile 0m 17s the patch passed
          +1 cc 0m 17s the patch passed
          +1 javac 0m 17s the patch passed
          -1 checkstyle 0m 10s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
          +1 mvnsite 0m 20s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 findbugs 0m 42s the patch passed
          +1 javadoc 0m 13s the patch passed
          +1 unit 0m 26s hadoop-yarn-server-common in the patch passed.
          +1 asflicense 0m 16s The patch does not generate ASF License warnings.
          13m 16s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12821710/YARN-3664-YARN-2915-v2.patch
          JIRA Issue YARN-3664
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux 2f6ebd4a847a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision YARN-2915 / 22db8fd
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12619/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/12619/artifact/patchprocess/whitespace-eol.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12619/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/12619/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 16s YARN-2915 passed +1 compile 0m 20s YARN-2915 passed +1 checkstyle 0m 13s YARN-2915 passed +1 mvnsite 0m 23s YARN-2915 passed +1 mvneclipse 0m 13s YARN-2915 passed +1 findbugs 0m 40s YARN-2915 passed +1 javadoc 0m 16s YARN-2915 passed +1 mvninstall 0m 18s the patch passed +1 compile 0m 17s the patch passed +1 cc 0m 17s the patch passed +1 javac 0m 17s the patch passed -1 checkstyle 0m 10s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) +1 mvnsite 0m 20s the patch passed +1 mvneclipse 0m 10s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 findbugs 0m 42s the patch passed +1 javadoc 0m 13s the patch passed +1 unit 0m 26s hadoop-yarn-server-common in the patch passed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 13m 16s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12821710/YARN-3664-YARN-2915-v2.patch JIRA Issue YARN-3664 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux 2f6ebd4a847a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision YARN-2915 / 22db8fd Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12619/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/12619/artifact/patchprocess/whitespace-eol.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12619/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common Console output https://builds.apache.org/job/PreCommit-YARN-Build/12619/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          subru Subru Krishnan added a comment -

          Varun Vasudev, thanks for your feedback.

          Attaching patch (v2) that incorporates your comments. Similar to YARN-5307, the names is definitely along the lines but are not exactly what you suggested as I have to tried to align with the final version of YARN-3662 which includes Vinod Kumar Vavilapalli/Wangda Tan's feedback too.

          Can we use something other than ByteBuffer for getParams - this'll become a problem if you ever expose this information via REST API or wish to update the object via a REST API(marshalling/unmarshalling ByteBuffer can be painful)

          I agree with your observation but we couldn't think of a better alternative based on the current understanding of the policy space (refer: YARN-5324/YARN-5325). Also since we have established this is an internal API, I feel we can revisit once the dust settles on the policies post testing. So I have left it as ByteBuffer for now.

          Show
          subru Subru Krishnan added a comment - Varun Vasudev , thanks for your feedback. Attaching patch (v2) that incorporates your comments. Similar to YARN-5307 , the names is definitely along the lines but are not exactly what you suggested as I have to tried to align with the final version of YARN-3662 which includes Vinod Kumar Vavilapalli / Wangda Tan 's feedback too. Can we use something other than ByteBuffer for getParams - this'll become a problem if you ever expose this information via REST API or wish to update the object via a REST API(marshalling/unmarshalling ByteBuffer can be painful) I agree with your observation but we couldn't think of a better alternative based on the current understanding of the policy space (refer: YARN-5324 / YARN-5325 ). Also since we have established this is an internal API, I feel we can revisit once the dust settles on the policies post testing. So I have left it as ByteBuffer for now.
          Hide
          vvasudev Varun Vasudev added a comment -

          Thanks for the patch Subru Krishnan! Some feedback -

          1)

          +  public FederationPolicyConfigurationGetResponse getPolicyConfigurationForQueue(
          +      FederationPolicyConfigurationGetRequest request) throws YarnException;
          

          Rename getPolicyConfigurationForQueue to getPolicyConfiguration

          2)

          +  public FederationPolicyConfigurationSetResponse setPolicyConfigurationForQueue(
          +      FederationPolicyConfigurationSetRequest request) throws YarnException;
          

          Rename setPolicyConfigurationForQueue to setPolicyConfiguration

          3)

          +  public Map<String, FederationPolicyConfiguration> getAllPolicies()
          +      throws YarnException;
          

          Rename getAllPolicies to getPoliciesConfigurations and the function should accept a GetPoliciesConfigurationsRequest and return a GetPoliciesConfigurationsResponse

          4)
          In FederationPolicyConfiguration, add an override for hashCode as well

          5)

          +  public abstract ByteBuffer getParams();
          

          Can we use something other than ByteBuffer for getParams - this'll become a problem if you ever expose this information via REST API or wish to update the object via a REST API(marshalling/unmarshalling ByteBuffer can be painful)

          6)
          Rename FederationPolicyConfigurationGetRequest to GetFederationPolicyConfigurationRequest

          7)
          Rename FederationPolicyConfigurationGetResponse to GetFederationPolicyConfigurationResponse

          8)

          +  @Public
          +  @Unstable
          +  public abstract FederationPolicyConfiguration getPolicy();
          +
          +  /**
          +   * Sets the policy configuration.
          +   *
          +   * @param policy the policy configuration for the specified queue
          +   */
          +  @Private
          +  @Unstable
          +  public abstract void setPolicy(FederationPolicyConfiguration policy);
          +
          +}
          

          Rename getPolicy and setPolicy to getPolicyConfiguration and setPolicyConfiguration

          9)
          Rename FederationPolicyConfigurationSetRequest to SetFederationPolicyConfigurationRequest

          10)

          +  /**
          +   * Get the policy configuration assigned to the queue.
          +   *
          +   * @return the policy for the specified queue
          +   */
          +  @Public
          +  @Unstable
          +  public abstract FederationPolicyConfiguration getPolicy();
          +
          +  /**
          +   * Set the policy configuration for the queue.
          +   *
          +   * @param policy the policy for the specified queue
          +   */
          +  @Private
          +  @Unstable
          +  public abstract void setPolicy(FederationPolicyConfiguration policy);
          

          Rename getPolicy and setPolicy to getPolicyConfiguration and setPolicyConfiguration

          11)
          Rename FederationPolicyConfigurationSetResponse to SetFederationPolicyConfigurationResponse

          Show
          vvasudev Varun Vasudev added a comment - Thanks for the patch Subru Krishnan ! Some feedback - 1) + public FederationPolicyConfigurationGetResponse getPolicyConfigurationForQueue( + FederationPolicyConfigurationGetRequest request) throws YarnException; Rename getPolicyConfigurationForQueue to getPolicyConfiguration 2) + public FederationPolicyConfigurationSetResponse setPolicyConfigurationForQueue( + FederationPolicyConfigurationSetRequest request) throws YarnException; Rename setPolicyConfigurationForQueue to setPolicyConfiguration 3) + public Map< String , FederationPolicyConfiguration> getAllPolicies() + throws YarnException; Rename getAllPolicies to getPoliciesConfigurations and the function should accept a GetPoliciesConfigurationsRequest and return a GetPoliciesConfigurationsResponse 4) In FederationPolicyConfiguration, add an override for hashCode as well 5) + public abstract ByteBuffer getParams(); Can we use something other than ByteBuffer for getParams - this'll become a problem if you ever expose this information via REST API or wish to update the object via a REST API(marshalling/unmarshalling ByteBuffer can be painful) 6) Rename FederationPolicyConfigurationGetRequest to GetFederationPolicyConfigurationRequest 7) Rename FederationPolicyConfigurationGetResponse to GetFederationPolicyConfigurationResponse 8) + @Public + @Unstable + public abstract FederationPolicyConfiguration getPolicy(); + + /** + * Sets the policy configuration. + * + * @param policy the policy configuration for the specified queue + */ + @Private + @Unstable + public abstract void setPolicy(FederationPolicyConfiguration policy); + +} Rename getPolicy and setPolicy to getPolicyConfiguration and setPolicyConfiguration 9) Rename FederationPolicyConfigurationSetRequest to SetFederationPolicyConfigurationRequest 10) + /** + * Get the policy configuration assigned to the queue. + * + * @ return the policy for the specified queue + */ + @Public + @Unstable + public abstract FederationPolicyConfiguration getPolicy(); + + /** + * Set the policy configuration for the queue. + * + * @param policy the policy for the specified queue + */ + @Private + @Unstable + public abstract void setPolicy(FederationPolicyConfiguration policy); Rename getPolicy and setPolicy to getPolicyConfiguration and setPolicyConfiguration 11) Rename FederationPolicyConfigurationSetResponse to SetFederationPolicyConfigurationResponse
          Hide
          subru Subru Krishnan added a comment -

          Updated patch (v1) that incorporates Wangda Tan's feedback:

          • Included FederationPolicyStore API class. I had missed it in previous patch, good catch.
          • Renamed record classes and updated Javadoc (with help from Carlo Curino) to make it more understandable.

          For more context, kindly refer to Carlo Curino's summary and associated policy patches in YARN-5323.

          Show
          subru Subru Krishnan added a comment - Updated patch (v1) that incorporates Wangda Tan 's feedback : Included FederationPolicyStore API class. I had missed it in previous patch, good catch. Renamed record classes and updated Javadoc (with help from Carlo Curino ) to make it more understandable. For more context, kindly refer to Carlo Curino 's summary and associated policy patches in YARN-5323 .
          Hide
          subru Subru Krishnan added a comment -

          Attaching v1 of the Federation PolicyStore APIs. The APIs provide for a generic mechanism to map the scheduler hierarchical queues across multiple YARN (sub)clusters.

          Please refer to the design doc in the parent JIRA (YARN-2915) for further details.

          The patch is available for review but is not ready for CI as it depends on YARN-3662.

          Show
          subru Subru Krishnan added a comment - Attaching v1 of the Federation PolicyStore APIs. The APIs provide for a generic mechanism to map the scheduler hierarchical queues across multiple YARN (sub)clusters. Please refer to the design doc in the parent JIRA ( YARN-2915 ) for further details. The patch is available for review but is not ready for CI as it depends on YARN-3662 .

            People

            • Assignee:
              subru Subru Krishnan
              Reporter:
              subru Subru Krishnan
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development