Hadoop YARN
  1. Hadoop YARN
  2. YARN-320

RM should always be able to renew its own tokens

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 3.0.0, 2.0.3-alpha, 0.23.6
    • Fix Version/s: 2.0.3-alpha, 0.23.6
    • Component/s: resourcemanager
    • Labels:
      None

      Description

      YARN-280 introduced fast-fail for job submissions with bad tokens. Unfortunately, other stack components like oozie and customers are acquiring RM tokens with a hardcoded dummy renewer value. These jobs would fail after 24 hours because the RM token couldn't be renewed, but fast-fail is failing them immediately. The RM should always be able to renew its own tokens submitted with a job. The renewer field may continue to specify an external user who can renew.

      1. YARN-320.branch-23.patch
        9 kB
        Daryn Sharp
      2. YARN-320.branch-23.patch
        3 kB
        Daryn Sharp
      3. YARN-320.patch
        8 kB
        Daryn Sharp

        Activity

        Hide
        Daryn Sharp added a comment -

        Proposed minimal change that fakes out the renewer. If the RM is talking to itself (RPC UGI matches login UGI), then it will claim the renewer is renewing the token.

        This also has the benefit of allowing the client to request a RM token that either it or a designated user can externally renew. Whereas today it's either the RM renews or someone else, but not both.

        It's been manually tested but I'll try to write unit tests and submit if this approach is ok.

        Show
        Daryn Sharp added a comment - Proposed minimal change that fakes out the renewer. If the RM is talking to itself (RPC UGI matches login UGI), then it will claim the renewer is renewing the token. This also has the benefit of allowing the client to request a RM token that either it or a designated user can externally renew. Whereas today it's either the RM renews or someone else, but not both. It's been manually tested but I'll try to write unit tests and submit if this approach is ok.
        Hide
        Siddharth Seth added a comment -

        Seems ok. Is the additional bit to extract the renewer from the token required ? Maybe we can avoid RPC all together for such tokens (issues by the same RM).

        Show
        Siddharth Seth added a comment - Seems ok. Is the additional bit to extract the renewer from the token required ? Maybe we can avoid RPC all together for such tokens (issues by the same RM).
        Hide
        Daryn Sharp added a comment -

        Admittedly, extracting the renewer is a bit dubious. It was the simplest way to satisfy the ADTSM checks w/o making changes to the core that would have a larger impact.

        In 1.x I tried to make the JT not do a lookback RPC to itself, but I don't recall why it was dinged. With the current design I'm not sure there's a good way for the token to get access to the RM's secret manager, but yes that would be ideal.

        Thanks, I'll wrap up the patch.

        Show
        Daryn Sharp added a comment - Admittedly, extracting the renewer is a bit dubious. It was the simplest way to satisfy the ADTSM checks w/o making changes to the core that would have a larger impact. In 1.x I tried to make the JT not do a lookback RPC to itself, but I don't recall why it was dinged. With the current design I'm not sure there's a good way for the token to get access to the RM's secret manager, but yes that would be ideal. Thanks, I'll wrap up the patch.
        Hide
        Daryn Sharp added a comment -

        Add unit tests, trunk patch forthcoming.

        Show
        Daryn Sharp added a comment - Add unit tests, trunk patch forthcoming.
        Hide
        Daryn Sharp added a comment -

        Patch for trunk and branch-2.

        Show
        Daryn Sharp added a comment - Patch for trunk and branch-2.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12563814/YARN-320.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-YARN-Build/325//testReport/
        Console output: https://builds.apache.org/job/PreCommit-YARN-Build/325//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12563814/YARN-320.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/325//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/325//console This message is automatically generated.
        Hide
        Siddharth Seth added a comment -

        +1. Committing this. Thanks Daryn.

        Show
        Siddharth Seth added a comment - +1. Committing this. Thanks Daryn.
        Hide
        Siddharth Seth added a comment -

        Committed to trunk, branch-2 and branch-23.

        Show
        Siddharth Seth added a comment - Committed to trunk, branch-2 and branch-23.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-trunk-Commit #3204 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3204/)
        YARN-320. RM should always be able to renew its own tokens. Contributed by Daryn Sharp (Revision 1431020)

        Result = SUCCESS
        sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1431020
        Files :

        • /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java
        • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java
        Show
        Hudson added a comment - Integrated in Hadoop-trunk-Commit #3204 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3204/ ) YARN-320 . RM should always be able to renew its own tokens. Contributed by Daryn Sharp (Revision 1431020) Result = SUCCESS sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1431020 Files : /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Yarn-trunk #92 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/92/)
        YARN-320. RM should always be able to renew its own tokens. Contributed by Daryn Sharp (Revision 1431020)

        Result = SUCCESS
        sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1431020
        Files :

        • /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java
        • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java
        Show
        Hudson added a comment - Integrated in Hadoop-Yarn-trunk #92 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/92/ ) YARN-320 . RM should always be able to renew its own tokens. Contributed by Daryn Sharp (Revision 1431020) Result = SUCCESS sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1431020 Files : /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #490 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/490/)
        YARN-320. RM should always be able to renew its own tokens. Contributed by Daryn Sharp (Revision 1431025)

        Result = FAILURE
        sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1431025
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-yarn-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java
        • /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #490 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/490/ ) YARN-320 . RM should always be able to renew its own tokens. Contributed by Daryn Sharp (Revision 1431025) Result = FAILURE sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1431025 Files : /hadoop/common/branches/branch-0.23/hadoop-yarn-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1309 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1309/)
        YARN-320. RM should always be able to renew its own tokens. Contributed by Daryn Sharp (Revision 1431020)

        Result = FAILURE
        sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1431020
        Files :

        • /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java
        • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1309 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1309/ ) YARN-320 . RM should always be able to renew its own tokens. Contributed by Daryn Sharp (Revision 1431020) Result = FAILURE sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1431020 Files : /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1281 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1281/)
        YARN-320. RM should always be able to renew its own tokens. Contributed by Daryn Sharp (Revision 1431020)

        Result = FAILURE
        sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1431020
        Files :

        • /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java
        • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1281 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1281/ ) YARN-320 . RM should always be able to renew its own tokens. Contributed by Daryn Sharp (Revision 1431020) Result = FAILURE sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1431020 Files : /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java

          People

          • Assignee:
            Daryn Sharp
            Reporter:
            Daryn Sharp
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development