Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-alpha2
    • Component/s: federation
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Two AMRMProxy bugs:

      First, the AMRMToken from RM should not be propagated to AM, since AMRMProxy will create a local AMRMToken for it.

      Second, the AMRMProxy Context is now parse the localAMRMTokenKeyId from amrmToken, but should be from localAmrmToken.

      1. YARN-6016.v1.patch
        3 kB
        Botong Huang
      2. YARN-6016.v2.patch
        7 kB
        Botong Huang
      3. YARN-6016.v3.patch
        9 kB
        Botong Huang

        Activity

        Hide
        botong Botong Huang added a comment -

        Thanks Subru and Jian!

        Show
        botong Botong Huang added a comment - Thanks Subru and Jian!
        Hide
        subru Subru Krishnan added a comment -

        Thanks Botong Huang for the patch and Jian He for the quick double-check. I committed this to trunk/branch-2.

        Show
        subru Subru Krishnan added a comment - Thanks Botong Huang for the patch and Jian He for the quick double-check. I committed this to trunk/branch-2.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #11130 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11130/)
        YARN-6016. Fix minor bugs in handling of local AMRMToken in AMRMProxy. (subru: rev 4d1f3d9020b8a8bf1d2a81e4d6ad20418ed9bcc2)

        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/TestAMRMProxyService.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/MockResourceManagerFacade.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyApplicationContextImpl.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAMRMProxy.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyService.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #11130 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11130/ ) YARN-6016 . Fix minor bugs in handling of local AMRMToken in AMRMProxy. (subru: rev 4d1f3d9020b8a8bf1d2a81e4d6ad20418ed9bcc2) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/TestAMRMProxyService.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/MockResourceManagerFacade.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyApplicationContextImpl.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAMRMProxy.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyService.java
        Hide
        jianhe Jian He added a comment -

        lgtm too, thanks

        Show
        jianhe Jian He added a comment - lgtm too, thanks
        Hide
        subru Subru Krishnan added a comment -

        Thanks Botong Huang for the clarification. +1 from side for the latest patch.

        I'll hold on till after the weekend in case anyone else (cc Jian He) wants to take a look and then commit.

        Show
        subru Subru Krishnan added a comment - Thanks Botong Huang for the clarification. +1 from side for the latest patch. I'll hold on till after the weekend in case anyone else (cc Jian He ) wants to take a look and then commit.
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 17s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
        0 mvndep 0m 45s Maven dependency ordering for branch
        +1 mvninstall 12m 49s trunk passed
        +1 compile 4m 57s trunk passed
        +1 checkstyle 0m 44s trunk passed
        +1 mvnsite 0m 57s trunk passed
        +1 mvneclipse 0m 39s trunk passed
        +1 findbugs 1m 19s trunk passed
        +1 javadoc 0m 43s trunk passed
        0 mvndep 0m 10s Maven dependency ordering for patch
        +1 mvninstall 0m 41s the patch passed
        +1 compile 4m 35s the patch passed
        +1 javac 4m 35s the patch passed
        +1 checkstyle 0m 41s the patch passed
        +1 mvnsite 0m 55s the patch passed
        +1 mvneclipse 0m 37s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 32s the patch passed
        +1 javadoc 0m 40s the patch passed
        +1 unit 12m 56s hadoop-yarn-server-nodemanager in the patch passed.
        +1 unit 16m 11s hadoop-yarn-client in the patch passed.
        +1 asflicense 0m 31s The patch does not generate ASF License warnings.
        70m 43s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue YARN-6016
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12847417/YARN-6016.v3.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 5f1c36e607e5 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / d3170f9
        Default Java 1.8.0_111
        findbugs v3.0.0
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14652/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/14652/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 45s Maven dependency ordering for branch +1 mvninstall 12m 49s trunk passed +1 compile 4m 57s trunk passed +1 checkstyle 0m 44s trunk passed +1 mvnsite 0m 57s trunk passed +1 mvneclipse 0m 39s trunk passed +1 findbugs 1m 19s trunk passed +1 javadoc 0m 43s trunk passed 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 0m 41s the patch passed +1 compile 4m 35s the patch passed +1 javac 4m 35s the patch passed +1 checkstyle 0m 41s the patch passed +1 mvnsite 0m 55s the patch passed +1 mvneclipse 0m 37s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 32s the patch passed +1 javadoc 0m 40s the patch passed +1 unit 12m 56s hadoop-yarn-server-nodemanager in the patch passed. +1 unit 16m 11s hadoop-yarn-client in the patch passed. +1 asflicense 0m 31s The patch does not generate ASF License warnings. 70m 43s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-6016 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12847417/YARN-6016.v3.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 5f1c36e607e5 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d3170f9 Default Java 1.8.0_111 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14652/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/14652/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        botong Botong Huang added a comment - - edited

        Hi Subru Krishnan, I've taken a look at TestAMRMProxy. There is a relevant test testE2ETokenRenewal. However, this test is really testing that AM can receive the new AMRMToken when AMRMProxy rolls its master key.

        Here in this Jira, the problem is between RM and AMRMProxy, when RM issues a new token,AMRMProxy should intercept it without forwarding it to AM. It is non-trivial to add it into TestAMRMProxy because both AMRMProxy and RM share the same config for rolling interval.

        I added tests in TestAMRMProxyService to cover this case. I also modified the test name and comments in TestAMRMProxy for clarification. Please see v3.patch

        Show
        botong Botong Huang added a comment - - edited Hi Subru Krishnan , I've taken a look at TestAMRMProxy . There is a relevant test testE2ETokenRenewal . However, this test is really testing that AM can receive the new AMRMToken when AMRMProxy rolls its master key. Here in this Jira, the problem is between RM and AMRMProxy, when RM issues a new token,AMRMProxy should intercept it without forwarding it to AM. It is non-trivial to add it into TestAMRMProxy because both AMRMProxy and RM share the same config for rolling interval. I added tests in TestAMRMProxyService to cover this case. I also modified the test name and comments in TestAMRMProxy for clarification. Please see v3.patch
        Hide
        subru Subru Krishnan added a comment -

        Thanks Botong Huang for the patch. Overall it looks good, I just had one request - can you add/update TestAMRMProxy as that was supposed to cover this scenario.

        Show
        subru Subru Krishnan added a comment - Thanks Botong Huang for the patch. Overall it looks good, I just had one request - can you add/update TestAMRMProxy as that was supposed to cover this scenario.
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 16s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        +1 mvninstall 13m 7s trunk passed
        +1 compile 0m 28s trunk passed
        +1 checkstyle 0m 16s trunk passed
        +1 mvnsite 0m 27s trunk passed
        +1 mvneclipse 0m 14s trunk passed
        +1 findbugs 0m 42s trunk passed
        +1 javadoc 0m 17s trunk passed
        +1 mvninstall 0m 22s the patch passed
        +1 compile 0m 24s the patch passed
        +1 javac 0m 24s the patch passed
        +1 checkstyle 0m 13s the patch passed
        +1 mvnsite 0m 23s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 0m 45s the patch passed
        +1 javadoc 0m 14s the patch passed
        +1 unit 12m 53s hadoop-yarn-server-nodemanager in the patch passed.
        +1 asflicense 0m 17s The patch does not generate ASF License warnings.
        32m 47s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue YARN-6016
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12846712/YARN-6016.v2.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 7da761c4fec3 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / e692316
        Default Java 1.8.0_111
        findbugs v3.0.0
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14632/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/14632/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 13m 7s trunk passed +1 compile 0m 28s trunk passed +1 checkstyle 0m 16s trunk passed +1 mvnsite 0m 27s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 0m 42s trunk passed +1 javadoc 0m 17s trunk passed +1 mvninstall 0m 22s the patch passed +1 compile 0m 24s the patch passed +1 javac 0m 24s the patch passed +1 checkstyle 0m 13s the patch passed +1 mvnsite 0m 23s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 45s the patch passed +1 javadoc 0m 14s the patch passed +1 unit 12m 53s hadoop-yarn-server-nodemanager in the patch passed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 32m 47s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-6016 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12846712/YARN-6016.v2.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 7da761c4fec3 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e692316 Default Java 1.8.0_111 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14632/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/14632/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.

          People

          • Assignee:
            botong Botong Huang
            Reporter:
            botong Botong Huang
          • Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development