Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha, 3.0.0, 0.23.7
    • Fix Version/s: 0.23.7, 2.0.4-alpha
    • Component/s: webhdfs
    • Labels:
      None

      Description

      Webhdfs will implicitly acquire a token to get/renew/cancel other tokens even though it neither needs nor uses the token. The implicit token triggers a renewer thread for the fs which is undesirable for daemons such as oozie and yarn's RM.

      1. HDFS-4567.patch
        12 kB
        Daryn Sharp
      2. HDFS-4567.patch
        12 kB
        Daryn Sharp
      3. HDFS-4567.branch-23.patch
        12 kB
        Daryn Sharp
      4. HDFS-4567.branch-23.patch
        12 kB
        Daryn Sharp

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1367 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1367/)
          HDFS-4567. Webhdfs does not need a token for token operations. Contributed by Daryn Sharp. (Revision 1454460)

          Result = SUCCESS
          kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454460
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsTokens.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsUrl.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1367 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1367/ ) HDFS-4567 . Webhdfs does not need a token for token operations. Contributed by Daryn Sharp. (Revision 1454460) Result = SUCCESS kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454460 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsTokens.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsUrl.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1339 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1339/)
          HDFS-4567. Webhdfs does not need a token for token operations. Contributed by Daryn Sharp. (Revision 1454460)

          Result = SUCCESS
          kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454460
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsTokens.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsUrl.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1339 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1339/ ) HDFS-4567 . Webhdfs does not need a token for token operations. Contributed by Daryn Sharp. (Revision 1454460) Result = SUCCESS kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454460 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsTokens.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsUrl.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #548 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/548/)
          Undo double application of patch to TestWebHdfsTokens.java for HDFS-4567. (Revision 1454474)
          HDFS-4567. Webhdfs does not need a token for token operations. Contributed by Daryn Sharp. (Revision 1454469)

          Result = SUCCESS
          kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454474
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsTokens.java

          kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454469
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsTokens.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsUrl.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #548 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/548/ ) Undo double application of patch to TestWebHdfsTokens.java for HDFS-4567 . (Revision 1454474) HDFS-4567 . Webhdfs does not need a token for token operations. Contributed by Daryn Sharp. (Revision 1454469) Result = SUCCESS kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454474 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsTokens.java kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454469 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsTokens.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsUrl.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Yarn-trunk #150 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/150/)
          HDFS-4567. Webhdfs does not need a token for token operations. Contributed by Daryn Sharp. (Revision 1454460)

          Result = SUCCESS
          kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454460
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsTokens.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsUrl.java
          Show
          Hudson added a comment - Integrated in Hadoop-Yarn-trunk #150 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/150/ ) HDFS-4567 . Webhdfs does not need a token for token operations. Contributed by Daryn Sharp. (Revision 1454460) Result = SUCCESS kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454460 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsTokens.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsUrl.java
          Hide
          Kihwal Lee added a comment -

          I've committed this to trunk, branch-2 and branch-0.23.

          Show
          Kihwal Lee added a comment - I've committed this to trunk, branch-2 and branch-0.23.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-trunk-Commit #3441 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3441/)
          HDFS-4567. Webhdfs does not need a token for token operations. Contributed by Daryn Sharp. (Revision 1454460)

          Result = SUCCESS
          kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454460
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsTokens.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsUrl.java
          Show
          Hudson added a comment - Integrated in Hadoop-trunk-Commit #3441 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3441/ ) HDFS-4567 . Webhdfs does not need a token for token operations. Contributed by Daryn Sharp. (Revision 1454460) Result = SUCCESS kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454460 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsTokens.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsUrl.java
          Hide
          Kihwal Lee added a comment -

          The patch looks good. The change prevents unconditional pre-acquisition of tokens, but do acquire a token for non-token related operations.

          There is slight change in behavior. If the file system instance is created, but no operation is done for long enough to expire tgt in the ticket cache, it won't be usable any more. Previously, pre-acquired token could outlast tgt and allow the file system to be used. But this change of behavior will only matter if ticket cache is used, not keytab and a WebHdfsFileSystem instance is created, but unused for a long time. Since services typically use keytab, this will mostly be confined to user interactive sessions and scripts where users need to make sure a valid tgt is present and the file system is used right away. Overall, there should be no visible negative effect.

          +1

          Show
          Kihwal Lee added a comment - The patch looks good. The change prevents unconditional pre-acquisition of tokens, but do acquire a token for non-token related operations. There is slight change in behavior. If the file system instance is created, but no operation is done for long enough to expire tgt in the ticket cache, it won't be usable any more. Previously, pre-acquired token could outlast tgt and allow the file system to be used. But this change of behavior will only matter if ticket cache is used, not keytab and a WebHdfsFileSystem instance is created, but unused for a long time. Since services typically use keytab, this will mostly be confined to user interactive sessions and scripts where users need to make sure a valid tgt is present and the file system is used right away. Overall, there should be no visible negative effect. +1
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12572769/HDFS-4567.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 tests included appear to have a timeout.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4059//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4059//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12572769/HDFS-4567.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 tests included appear to have a timeout. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4059//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4059//console This message is automatically generated.
          Hide
          Daryn Sharp added a comment -

          Oops, same patches with just the missing timeouts added.

          Show
          Daryn Sharp added a comment - Oops, same patches with just the missing timeouts added.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12572621/HDFS-4567.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          -1 one of tests included doesn't have a timeout.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4052//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4052//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12572621/HDFS-4567.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. -1 one of tests included doesn't have a timeout. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4052//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4052//console This message is automatically generated.
          Hide
          Daryn Sharp added a comment -

          If the token is not in the UGI, then it's only fetched on demand for the first non-token operation.

          Additionally verified patch via fetchdt. Token get/renew/cancel no longer get a token before performing the operation.

          Show
          Daryn Sharp added a comment - If the token is not in the UGI, then it's only fetched on demand for the first non-token operation. Additionally verified patch via fetchdt. Token get/renew/cancel no longer get a token before performing the operation.

            People

            • Assignee:
              Daryn Sharp
              Reporter:
              Daryn Sharp
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development