Hadoop Common
  1. Hadoop Common
  2. HADOOP-6994

Api to get delegation token in AbstractFileSystem

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.23.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      APIs to get delegation tokens is required in AbstractFileSystem. AbstractFileSystems are accessed via file context therefore an API to get list of AbstractFileSystems accessed in a path is also needed.
      A path may refer to several file systems and delegation tokens could be needed for many of them for a client to be able to successfully access the path.

      1. HADOOP-6994.9.yahoo-merge.patch
        13 kB
        Jitendra Nath Pandey
      2. HADOOP-6994.9.patch
        13 kB
        Jitendra Nath Pandey
      3. HADOOP-6994.8.patch
        13 kB
        Jitendra Nath Pandey
      4. HADOOP-6994.5.patch
        12 kB
        Jitendra Nath Pandey
      5. HADOOP-6994.4.patch
        9 kB
        Jitendra Nath Pandey
      6. HADOOP-6994.2.patch
        9 kB
        Jitendra Nath Pandey
      7. HADOOP-6994.1.patch
        9 kB
        Jitendra Nath Pandey

        Issue Links

          Activity

          Hide
          Tom White added a comment -

          The new methods on AbstractFileSystem are public, yet Token and AbstractDelegationTokenIdentifier are LimitedPrivate, which seems inconsistent.

          Show
          Tom White added a comment - The new methods on AbstractFileSystem are public, yet Token and AbstractDelegationTokenIdentifier are LimitedPrivate, which seems inconsistent.
          Hide
          Jakob Homan added a comment -

          The new methods on AbstractFileSystem are public, yet Token and AbstractDelegationTokenIdentifier are LimitedPrivate, which seems inconsistent.

          I also have this concern, particularly in regard to HADOOP-6988.

          Show
          Jakob Homan added a comment - The new methods on AbstractFileSystem are public, yet Token and AbstractDelegationTokenIdentifier are LimitedPrivate, which seems inconsistent. I also have this concern, particularly in regard to HADOOP-6988 .
          Hide
          Jitendra Nath Pandey added a comment -

          New patch addressing the comment.

          The InterfaceAudience for the new APIs are annotated to be LimitedPrivate.
          The getDelegationToken API in FileSystem is also changed to LimitedPrivate.

          Show
          Jitendra Nath Pandey added a comment - New patch addressing the comment. The InterfaceAudience for the new APIs are annotated to be LimitedPrivate. The getDelegationToken API in FileSystem is also changed to LimitedPrivate.
          Hide
          Jitendra Nath Pandey added a comment -

          This patch is based against the latest trunk.

          Show
          Jitendra Nath Pandey added a comment - This patch is based against the latest trunk.
          Hide
          Jitendra Nath Pandey added a comment -

          New patch adds the getDelegationTokens API in FileContext and the API in AFS is changed to return a list of tokens.

          Show
          Jitendra Nath Pandey added a comment - New patch adds the getDelegationTokens API in FileContext and the API in AFS is changed to return a list of tokens.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12475754/HADOOP-6994.5.patch
          against trunk revision 1090039.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.fs.TestFilterFileSystem
          org.apache.hadoop.fs.TestFilterFs

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/335//testReport/
          Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/335//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/335//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12475754/HADOOP-6994.5.patch against trunk revision 1090039. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.fs.TestFilterFileSystem org.apache.hadoop.fs.TestFilterFs +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/335//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/335//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/335//console This message is automatically generated.
          Hide
          Suresh Srinivas added a comment -
          1. AbstractFileSystem.java
          2. #getDelegationTokens() - Javadoc says "Get a new delegation token for this file system.". It would be good to explain it as "Get one or more delegations tokens associated with the filesystem. Normally a file system returns a single delegation token. A file system that manages multiple file systems underneath, could return set of delegation tokens for all the file systems it manages."This comment applies to FileSystem also.
          3. When a file system returns multiple delegation tokens, how does renew and cancel work? Woudl the file system know where to renew/cancel the token?
          Show
          Suresh Srinivas added a comment - AbstractFileSystem.java #getDelegationTokens() - Javadoc says "Get a new delegation token for this file system.". It would be good to explain it as "Get one or more delegations tokens associated with the filesystem. Normally a file system returns a single delegation token. A file system that manages multiple file systems underneath, could return set of delegation tokens for all the file systems it manages."This comment applies to FileSystem also. When a file system returns multiple delegation tokens, how does renew and cancel work? Woudl the file system know where to renew/cancel the token?
          Hide
          Jitendra Nath Pandey added a comment -

          2. OK
          3. For renew and cancel, a file system that manages multiple filesystems underneath should pick the relevant filesystem by matching the service set in the token.

          Show
          Jitendra Nath Pandey added a comment - 2. OK 3. For renew and cancel, a file system that manages multiple filesystems underneath should pick the relevant filesystem by matching the service set in the token.
          Hide
          Suresh Srinivas added a comment -

          +1 for the patch.

          Show
          Suresh Srinivas added a comment - +1 for the patch.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12475754/HADOOP-6994.5.patch
          against trunk revision 1090485.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.fs.TestFilterFileSystem
          org.apache.hadoop.fs.TestFilterFs

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/341//testReport/
          Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/341//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/341//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12475754/HADOOP-6994.5.patch against trunk revision 1090485. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.fs.TestFilterFileSystem org.apache.hadoop.fs.TestFilterFs +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/341//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/341//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/341//console This message is automatically generated.
          Hide
          Jitendra Nath Pandey added a comment -

          The latest patch fixes FilterFileSystem and FilterFS. The renew and cancel delegation token APIs are removed from the patch. These APIs should be implemented as a util methods in SecurityUtil and the filesystem to renew or cancel can be figured out from the service and kind set in the token itself. Implementation in SecurityUtil will be taken up in a separate jira.

          Show
          Jitendra Nath Pandey added a comment - The latest patch fixes FilterFileSystem and FilterFS. The renew and cancel delegation token APIs are removed from the patch. These APIs should be implemented as a util methods in SecurityUtil and the filesystem to renew or cancel can be figured out from the service and kind set in the token itself. Implementation in SecurityUtil will be taken up in a separate jira.
          Hide
          Suresh Srinivas added a comment -

          +1 for the change.

          Show
          Suresh Srinivas added a comment - +1 for the change.
          Hide
          Jitendra Nath Pandey added a comment -

          Updated patch includes a javadoc for the new unit test. Rest is all identical with previous patch.

          Show
          Jitendra Nath Pandey added a comment - Updated patch includes a javadoc for the new unit test. Rest is all identical with previous patch.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12476162/HADOOP-6994.9.patch
          against trunk revision 1090485.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/343//testReport/
          Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/343//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/343//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12476162/HADOOP-6994.9.patch against trunk revision 1090485. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/343//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/343//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/343//console This message is automatically generated.
          Hide
          Jitendra Nath Pandey added a comment -

          I have committed this.

          Show
          Jitendra Nath Pandey added a comment - I have committed this.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #549 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/549/)
          HADOOP-6994. Api to get delegation token in AbstractFileSystem. Contributed by jitendra.

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #549 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/549/ ) HADOOP-6994 . Api to get delegation token in AbstractFileSystem. Contributed by jitendra.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk #660 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/660/)
          HADOOP-6994. Api to get delegation token in AbstractFileSystem. Contributed by jitendra.

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk #660 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/660/ ) HADOOP-6994 . Api to get delegation token in AbstractFileSystem. Contributed by jitendra.
          Hide
          Jitendra Nath Pandey added a comment -

          Patch for yahoo-merge branch.

          Show
          Jitendra Nath Pandey added a comment - Patch for yahoo-merge branch.

            People

            • Assignee:
              Jitendra Nath Pandey
              Reporter:
              Jitendra Nath Pandey
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development