Details

    • Hadoop Flags:
      Reviewed

      Description

      The Federation Application State encapsulates the mapping between an application and it's home sub-cluster, i.e. the sub-cluster to which it is submitted to by the Router. Please refer to the design doc in parent JIRA for further details.

      1. YARN-5307-YARN-2915-v1.patch
        69 kB
        Subru Krishnan
      2. YARN-5307-YARN-2915-v2.patch
        62 kB
        Subru Krishnan
      3. YARN-5307-YARN-2915-v3.patch
        103 kB
        Subru Krishnan
      4. YARN-5307-YARN-2915-v4.patch
        104 kB
        Subru Krishnan
      5. YARN-5307-YARN-2915-v5.patch
        100 kB
        Subru Krishnan

        Issue Links

          Activity

          Hide
          subru Subru Krishnan added a comment -

          I just committed this to branch YARN-2915. Thanks Varun Vasudev and Wangda Tan for the diligent reviews!

          Show
          subru Subru Krishnan added a comment - I just committed this to branch YARN-2915 . Thanks Varun Vasudev and Wangda Tan for the diligent reviews!
          Hide
          leftnoteasy Wangda Tan added a comment -

          +1, thanks Subru Krishnan

          Show
          leftnoteasy Wangda Tan added a comment - +1, thanks Subru Krishnan
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 12s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 31s YARN-2915 passed
          +1 compile 0m 23s YARN-2915 passed
          +1 checkstyle 0m 14s YARN-2915 passed
          +1 mvnsite 0m 26s YARN-2915 passed
          +1 mvneclipse 0m 14s YARN-2915 passed
          +1 findbugs 0m 45s YARN-2915 passed
          +1 javadoc 0m 18s YARN-2915 passed
          +1 mvninstall 0m 27s the patch passed
          +1 compile 0m 20s the patch passed
          +1 cc 0m 20s the patch passed
          +1 javac 0m 20s the patch passed
          +1 checkstyle 0m 11s the patch passed
          +1 mvnsite 0m 24s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 55s the patch passed
          +1 javadoc 0m 15s the patch passed
          +1 unit 0m 33s hadoop-yarn-server-common in the patch passed.
          +1 asflicense 0m 16s The patch does not generate ASF License warnings.
          15m 21s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12821953/YARN-5307-YARN-2915-v5.patch
          JIRA Issue YARN-5307
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux ba80aa7ac143 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision YARN-2915 / 22db8fd
          Default Java 1.8.0_101
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12639/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/12639/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 31s YARN-2915 passed +1 compile 0m 23s YARN-2915 passed +1 checkstyle 0m 14s YARN-2915 passed +1 mvnsite 0m 26s YARN-2915 passed +1 mvneclipse 0m 14s YARN-2915 passed +1 findbugs 0m 45s YARN-2915 passed +1 javadoc 0m 18s YARN-2915 passed +1 mvninstall 0m 27s the patch passed +1 compile 0m 20s the patch passed +1 cc 0m 20s the patch passed +1 javac 0m 20s the patch passed +1 checkstyle 0m 11s the patch passed +1 mvnsite 0m 24s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 55s the patch passed +1 javadoc 0m 15s the patch passed +1 unit 0m 33s hadoop-yarn-server-common in the patch passed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 15m 21s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12821953/YARN-5307-YARN-2915-v5.patch JIRA Issue YARN-5307 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux ba80aa7ac143 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision YARN-2915 / 22db8fd Default Java 1.8.0_101 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12639/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common Console output https://builds.apache.org/job/PreCommit-YARN-Build/12639/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          subru Subru Krishnan added a comment -

          Thanks Wangda Tan for your feedback. I have addressed all of them in v5.

          Show
          subru Subru Krishnan added a comment - Thanks Wangda Tan for your feedback. I have addressed all of them in v5.
          Hide
          subru Subru Krishnan added a comment -

          Thanks Varun Vasudev for reviewing the patch.

          I'll take care of it as soon as Wangda Tan signs off.

          Show
          subru Subru Krishnan added a comment - Thanks Varun Vasudev for reviewing the patch. I'll take care of it as soon as Wangda Tan signs off.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Hi Subru Krishnan,

          Few comments:

          • For the responsibility of FederationApplicationStateStore, what else state you plan to add for applications? What I can see is it only manages home-sub-cluster of applications, renaming it to FederationApplicationHomeSubClusterStore if no more items are planned?
          • For ApplicationHomeSubClusterMap and all related response / request / method name, remove "map" from these names? I was trying to find map-like data structure in definition..
          • Cannot figure out who will use FederationStateStore and how it will be used, should we move it to more related JIRA?
          Show
          leftnoteasy Wangda Tan added a comment - Hi Subru Krishnan , Few comments: For the responsibility of FederationApplicationStateStore, what else state you plan to add for applications? What I can see is it only manages home-sub-cluster of applications, renaming it to FederationApplicationHomeSubClusterStore if no more items are planned? For ApplicationHomeSubClusterMap and all related response / request / method name, remove "map" from these names? I was trying to find map-like data structure in definition.. Cannot figure out who will use FederationStateStore and how it will be used, should we move it to more related JIRA?
          Hide
          vvasudev Varun Vasudev added a comment -

          +1. Let me know if I should commit or you can take care of it.

          Show
          vvasudev Varun Vasudev added a comment - +1. Let me know if I should commit or you can take care of it.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 7s YARN-2915 passed
          +1 compile 0m 20s YARN-2915 passed
          +1 checkstyle 0m 13s YARN-2915 passed
          +1 mvnsite 0m 22s YARN-2915 passed
          +1 mvneclipse 0m 14s YARN-2915 passed
          +1 findbugs 0m 38s YARN-2915 passed
          +1 javadoc 0m 15s YARN-2915 passed
          +1 mvninstall 0m 19s the patch passed
          +1 compile 0m 17s the patch passed
          +1 cc 0m 17s the patch passed
          +1 javac 0m 17s the patch passed
          +1 checkstyle 0m 10s the patch passed
          +1 mvnsite 0m 20s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 44s the patch passed
          +1 javadoc 0m 13s the patch passed
          +1 unit 0m 26s hadoop-yarn-server-common in the patch passed.
          +1 asflicense 0m 16s The patch does not generate ASF License warnings.
          13m 2s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12821681/YARN-5307-YARN-2915-v4.patch
          JIRA Issue YARN-5307
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux 857cc5bf8d72 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision YARN-2915 / 22db8fd
          Default Java 1.8.0_101
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12616/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/12616/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 7s YARN-2915 passed +1 compile 0m 20s YARN-2915 passed +1 checkstyle 0m 13s YARN-2915 passed +1 mvnsite 0m 22s YARN-2915 passed +1 mvneclipse 0m 14s YARN-2915 passed +1 findbugs 0m 38s YARN-2915 passed +1 javadoc 0m 15s YARN-2915 passed +1 mvninstall 0m 19s the patch passed +1 compile 0m 17s the patch passed +1 cc 0m 17s the patch passed +1 javac 0m 17s the patch passed +1 checkstyle 0m 10s the patch passed +1 mvnsite 0m 20s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 44s the patch passed +1 javadoc 0m 13s the patch passed +1 unit 0m 26s hadoop-yarn-server-common in the patch passed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 13m 2s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12821681/YARN-5307-YARN-2915-v4.patch JIRA Issue YARN-5307 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux 857cc5bf8d72 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision YARN-2915 / 22db8fd Default Java 1.8.0_101 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12616/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common Console output https://builds.apache.org/job/PreCommit-YARN-Build/12616/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          subru Subru Krishnan added a comment -

          Updated patch (v4) with minor typo fixes to private methods

          Show
          subru Subru Krishnan added a comment - Updated patch (v4) with minor typo fixes to private methods
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 10m 16s YARN-2915 passed
          +1 compile 0m 19s YARN-2915 passed
          +1 checkstyle 0m 15s YARN-2915 passed
          +1 mvnsite 0m 24s YARN-2915 passed
          +1 mvneclipse 0m 17s YARN-2915 passed
          +1 findbugs 0m 42s YARN-2915 passed
          +1 javadoc 0m 16s YARN-2915 passed
          +1 mvninstall 0m 19s the patch passed
          +1 compile 0m 17s the patch passed
          +1 cc 0m 17s the patch passed
          +1 javac 0m 17s the patch passed
          +1 checkstyle 0m 9s the patch passed
          +1 mvnsite 0m 19s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 45s the patch passed
          +1 javadoc 0m 13s the patch passed
          +1 unit 0m 26s hadoop-yarn-server-common in the patch passed.
          +1 asflicense 0m 16s The patch does not generate ASF License warnings.
          16m 18s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12821484/YARN-5307-YARN-2915-v3.patch
          JIRA Issue YARN-5307
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux 13cf4b6dc4f4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision YARN-2915 / 22db8fd
          Default Java 1.8.0_101
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12604/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/12604/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 10m 16s YARN-2915 passed +1 compile 0m 19s YARN-2915 passed +1 checkstyle 0m 15s YARN-2915 passed +1 mvnsite 0m 24s YARN-2915 passed +1 mvneclipse 0m 17s YARN-2915 passed +1 findbugs 0m 42s YARN-2915 passed +1 javadoc 0m 16s YARN-2915 passed +1 mvninstall 0m 19s the patch passed +1 compile 0m 17s the patch passed +1 cc 0m 17s the patch passed +1 javac 0m 17s the patch passed +1 checkstyle 0m 9s the patch passed +1 mvnsite 0m 19s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 45s the patch passed +1 javadoc 0m 13s the patch passed +1 unit 0m 26s hadoop-yarn-server-common in the patch passed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 16m 18s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12821484/YARN-5307-YARN-2915-v3.patch JIRA Issue YARN-5307 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux 13cf4b6dc4f4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision YARN-2915 / 22db8fd Default Java 1.8.0_101 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12604/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common Console output https://builds.apache.org/job/PreCommit-YARN-Build/12604/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          subru Subru Krishnan added a comment -

          Varun Vasudev, thanks for your feedback.

          Attaching patch (v3) that incorporates your comment. The names may not be exactly what you suggested as I have to tried to align with the final version of YARN-3662 which includes Vinod Kumar Vavilapalli/Wangda Tan's feedback too.

          Show
          subru Subru Krishnan added a comment - Varun Vasudev , thanks for your feedback. Attaching patch (v3) that incorporates your comment. The names may not be exactly what you suggested as I have to tried to align with the final version of YARN-3662 which includes Vinod Kumar Vavilapalli / Wangda Tan 's feedback too.
          Hide
          vvasudev Varun Vasudev added a comment -

          Few more renames -

          Rename FederationInsertNewApplicationRequest to AddFederationApplicationRequest
          Rename FederationInsertNewApplicationResponse to AddFederationApplicationResponse
          Rename FederationUpdateApplicationRequest to UpdateFederationApplicationRequest
          Rename FederationUpdateApplicationResponse to UpdateFederationApplicationResponse
          Rename FederationDeleteApplicationRequest to DeleteFederationApplicationRequest
          Rename FederationDeleteApplicationResponse to DeleteFederationApplicationResponse

          Show
          vvasudev Varun Vasudev added a comment - Few more renames - Rename FederationInsertNewApplicationRequest to AddFederationApplicationRequest Rename FederationInsertNewApplicationResponse to AddFederationApplicationResponse Rename FederationUpdateApplicationRequest to UpdateFederationApplicationRequest Rename FederationUpdateApplicationResponse to UpdateFederationApplicationResponse Rename FederationDeleteApplicationRequest to DeleteFederationApplicationRequest Rename FederationDeleteApplicationResponse to DeleteFederationApplicationResponse
          Hide
          vvasudev Varun Vasudev added a comment -

          Forgot to mention one more thing - maybe the API classes should go into hadoop-yarn-api?

          Show
          vvasudev Varun Vasudev added a comment - Forgot to mention one more thing - maybe the API classes should go into hadoop-yarn-api?
          Hide
          vvasudev Varun Vasudev added a comment -

          Thanks for the patch Subru Krishnan! My thoughts -

          1)

          +  public FederationInsertNewApplicationResponse insertNewApplication(
          +      FederationInsertNewApplicationRequest request) throws YarnException; 
          

          Rename insertNewApplication to addApplication

          2)

          +  public FederationApplicationInfo getApplicationInfo(ApplicationId appId)
          +      throws YarnException;
           

          Rename getApplicationInfo to getApplication
          In addition, the function should return a GetApplicationResponse and accept GetApplicationRequest

          3)

          +  public Map<ApplicationId, FederationApplicationInfo> getAllApplicationsInfo()
          +      throws YarnException;
          

          Rename getAllApplicationsInfo to getApplications. Like above, it should return a GetApplicationsResponse and accept GetApplicationsRequest

          4)

          +  @Override
          +  public int hashCode() {
          +    return getApplicationId().hashCode();
          +  }
          

          In FederationApplicationInfo the hashCode function should take into account the same params as the equals function.

          Show
          vvasudev Varun Vasudev added a comment - Thanks for the patch Subru Krishnan ! My thoughts - 1) + public FederationInsertNewApplicationResponse insertNewApplication( + FederationInsertNewApplicationRequest request) throws YarnException; Rename insertNewApplication to addApplication 2) + public FederationApplicationInfo getApplicationInfo(ApplicationId appId) + throws YarnException; Rename getApplicationInfo to getApplication In addition, the function should return a GetApplicationResponse and accept GetApplicationRequest 3) + public Map<ApplicationId, FederationApplicationInfo> getAllApplicationsInfo() + throws YarnException; Rename getAllApplicationsInfo to getApplications. Like above, it should return a GetApplicationsResponse and accept GetApplicationsRequest 4) + @Override + public int hashCode() { + return getApplicationId().hashCode(); + } In FederationApplicationInfo the hashCode function should take into account the same params as the equals function.
          Hide
          subru Subru Krishnan added a comment -

          Updated patch (v2) that incorporates Wangda Tan's feedback as described here.

          Show
          subru Subru Krishnan added a comment - Updated patch (v2) that incorporates Wangda Tan 's feedback as described here .
          Hide
          subru Subru Krishnan added a comment -

          Uploading v1 of the FederationApplicationState APIs.

          The patch is available for review but is not ready for CI as it depends on YARN-3662. I'll add the tests for the app state protocol records post commit of YARN-3662.

          Show
          subru Subru Krishnan added a comment - Uploading v1 of the FederationApplicationState APIs. The patch is available for review but is not ready for CI as it depends on YARN-3662 . I'll add the tests for the app state protocol records post commit of YARN-3662 .

            People

            • Assignee:
              subru Subru Krishnan
              Reporter:
              subru Subru Krishnan
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development