Hadoop YARN
  1. Hadoop YARN
  2. YARN-360

Allow apps to concurrently register tokens for renewal

        Details

        • Type: Bug Bug
        • Status: Closed
        • Priority: Critical Critical
        • Resolution: Fixed
        • Affects Version/s: 0.23.3, 3.0.0, 2.0.0-alpha
        • Fix Version/s: 2.0.3-alpha, 0.23.7
        • Component/s: None
        • Labels:
          None

          Description

          DelegationTokenRenewer#addApplication has an unnecessary synchronized keyword. This serializes job submissions and can add unnecessary latency and/or hang all submissions if there are problems renewing the token.

          1. Text File
            YARN-357.patch
            5 kB
            Daryn Sharp
          2. Text File
            YARN-360.patch
            4 kB
            Daryn Sharp

            Issue Links

            is required by

            Bug - A problem which impairs or prevents the functions of the product. YARN-357 App submission should not be synchronized

            • Major - Major loss of function.
            • Closed

              Activity

              Descending order - Click to sort in ascending order
              Hide
              Permalink
              Hudson added a comment -

              Integrated in Hadoop-Mapreduce-trunk #1335 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1335/)
              YARN-360. Allow apps to concurrently register tokens for renewal. Contributed by Daryn Sharp. (Revision 1442441)

              Result = SUCCESS
              sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1442441
              Files :

              • /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
              • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml
              • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/DelegationTokenRenewer.java
              • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestDelegationTokenRenewer.java
              Show
              Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1335 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1335/ ) YARN-360 . Allow apps to concurrently register tokens for renewal. Contributed by Daryn Sharp. (Revision 1442441) Result = SUCCESS sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1442441 Files : /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/DelegationTokenRenewer.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestDelegationTokenRenewer.java
              Hide
              Permalink
              Hudson added a comment -

              Integrated in Hadoop-Hdfs-trunk #1307 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1307/)
              YARN-360. Allow apps to concurrently register tokens for renewal. Contributed by Daryn Sharp. (Revision 1442441)

              Result = FAILURE
              sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1442441
              Files :

              • /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
              • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml
              • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/DelegationTokenRenewer.java
              • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestDelegationTokenRenewer.java
              Show
              Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1307 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1307/ ) YARN-360 . Allow apps to concurrently register tokens for renewal. Contributed by Daryn Sharp. (Revision 1442441) Result = FAILURE sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1442441 Files : /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/DelegationTokenRenewer.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestDelegationTokenRenewer.java
              Hide
              Permalink
              Hudson added a comment -

              Integrated in Hadoop-Hdfs-0.23-Build #516 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/516/)
              YARN-360. Allow apps to concurrently register tokens for renewal. Contributed by Daryn Sharp. (Revision 1442443)

              Result = SUCCESS
              sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1442443
              Files :

              • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
              • /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml
              • /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/DelegationTokenRenewer.java
              • /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestDelegationTokenRenewer.java
              Show
              Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #516 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/516/ ) YARN-360 . Allow apps to concurrently register tokens for renewal. Contributed by Daryn Sharp. (Revision 1442443) Result = SUCCESS sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1442443 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/DelegationTokenRenewer.java /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestDelegationTokenRenewer.java
              Hide
              Permalink
              Hudson added a comment -

              Integrated in Hadoop-Yarn-trunk #118 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/118/)
              YARN-360. Allow apps to concurrently register tokens for renewal. Contributed by Daryn Sharp. (Revision 1442441)

              Result = FAILURE
              sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1442441
              Files :

              • /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
              • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml
              • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/DelegationTokenRenewer.java
              • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestDelegationTokenRenewer.java
              Show
              Hudson added a comment - Integrated in Hadoop-Yarn-trunk #118 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/118/ ) YARN-360 . Allow apps to concurrently register tokens for renewal. Contributed by Daryn Sharp. (Revision 1442441) Result = FAILURE sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1442441 Files : /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/DelegationTokenRenewer.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestDelegationTokenRenewer.java
              Hide
              Permalink
              Siddharth Seth added a comment -

              Committed to trunk, branch-2 and branch-0.23. Thanks Daryn!

              Show
              Siddharth Seth added a comment - Committed to trunk, branch-2 and branch-0.23. Thanks Daryn!
              Hide
              Permalink
              Hudson added a comment -

              Integrated in Hadoop-trunk-Commit #3320 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3320/)
              YARN-360. Allow apps to concurrently register tokens for renewal. Contributed by Daryn Sharp. (Revision 1442441)

              Result = SUCCESS
              sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1442441
              Files :

              • /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
              • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml
              • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/DelegationTokenRenewer.java
              • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestDelegationTokenRenewer.java
              Show
              Hudson added a comment - Integrated in Hadoop-trunk-Commit #3320 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3320/ ) YARN-360 . Allow apps to concurrently register tokens for renewal. Contributed by Daryn Sharp. (Revision 1442441) Result = SUCCESS sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1442441 Files : /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/DelegationTokenRenewer.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestDelegationTokenRenewer.java
              Hide
              Permalink
              Siddharth Seth added a comment -

              +1. Committing this. Nice unit test btw. The test failure isn't related; passes locally on multiple runs.

              Show
              Siddharth Seth added a comment - +1. Committing this. Nice unit test btw. The test failure isn't related; passes locally on multiple runs.
              Hide
              Permalink
              Hadoop QA added a comment -

              -1 overall. Here are the results of testing the latest attachment
              http://issues.apache.org/jira/secure/attachment/12566781/YARN-357.patch
              against trunk revision .

              +1 @author. The patch does not contain any @author tags.

              +1 tests included. The patch appears to include 1 new or modified test files.

              +1 javac. The applied patch does not increase the total number of javac compiler warnings.

              +1 javadoc. The javadoc tool did not generate any warning messages.

              +1 eclipse:eclipse. The patch built with eclipse:eclipse.

              +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

              +1 release audit. The applied patch does not increase the total number of release audit warnings.

              -1 core tests. The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

              org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServices

              +1 contrib tests. The patch passed contrib unit tests.

              Test results: https://builds.apache.org/job/PreCommit-YARN-Build/371//testReport/
              Console output: https://builds.apache.org/job/PreCommit-YARN-Build/371//console

              This message is automatically generated.

              Show
              Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12566781/YARN-357.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServices +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/371//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/371//console This message is automatically generated.
              Hide
              Permalink
              Daryn Sharp added a comment -

              Update findbugs exclusions for false positive on a thread-safe Timer. TestRMWebServices still fails even when I revert this patch.

              Show
              Daryn Sharp added a comment - Update findbugs exclusions for false positive on a thread-safe Timer. TestRMWebServices still fails even when I revert this patch.
              Hide
              Permalink
              Daryn Sharp added a comment -

              Findbugs is false positive. Timer class is thread-safe, but it's instantiated and cancelled in sync'ed methods so findbugs is assuming that scheduling tasks should be sync'ed too.

              Test failure appears unrelated, but looking into it.

              Show
              Daryn Sharp added a comment - Findbugs is false positive. Timer class is thread-safe, but it's instantiated and cancelled in sync'ed methods so findbugs is assuming that scheduling tasks should be sync'ed too. Test failure appears unrelated, but looking into it.
              Hide
              Permalink
              Hadoop QA added a comment -

              -1 overall. Here are the results of testing the latest attachment
              http://issues.apache.org/jira/secure/attachment/12566535/YARN-360.patch
              against trunk revision .

              +1 @author. The patch does not contain any @author tags.

              +1 tests included. The patch appears to include 1 new or modified test files.

              +1 javac. The applied patch does not increase the total number of javac compiler warnings.

              +1 javadoc. The javadoc tool did not generate any warning messages.

              +1 eclipse:eclipse. The patch built with eclipse:eclipse.

              -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

              +1 release audit. The applied patch does not increase the total number of release audit warnings.

              -1 core tests. The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

              org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServices

              +1 contrib tests. The patch passed contrib unit tests.

              Test results: https://builds.apache.org/job/PreCommit-YARN-Build/367//testReport/
              Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/367//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
              Console output: https://builds.apache.org/job/PreCommit-YARN-Build/367//console

              This message is automatically generated.

              Show
              Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12566535/YARN-360.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServices +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/367//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/367//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html Console output: https://builds.apache.org/job/PreCommit-YARN-Build/367//console This message is automatically generated.

                People

                • Assignee:
                  Daryn Sharp
                  Reporter:
                  Daryn Sharp
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  7 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Development