Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-994

Provide methods for obtaining delegation token from Namenode for hftp and other uses

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None

      Description

      In hftp, destination clusters will require an RPC-version-agnostic means of obtaining delegation tokens from the source cluster. The easiest method is provide a webservice to retrieve a token over http. This can be encrypted via SSL (backed by Kerberos, done in another JIRA), providing security for cross-cluster hftp operations.

      1. HDFS-994.patch
        15 kB
        Jakob Homan
      2. HDFS-994-0_20.1.patch
        17 kB
        Jitendra Nath Pandey
      3. HDFS-994-2.patch
        15 kB
        Jakob Homan
      4. HDFS-994-3.patch
        16 kB
        Jakob Homan
      5. HDFS-994-4.patch
        16 kB
        Jakob Homan
      6. HDFS-994-5.patch
        16 kB
        Jakob Homan

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #275 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/275/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #275 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/275/ )
          Hide
          Hudson added a comment -

          Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #302 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/302/)

          Show
          Hudson added a comment - Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #302 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/302/ )
          Hide
          Hudson added a comment -

          Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #146 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/146/)

          Show
          Hudson added a comment - Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #146 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/146/ )
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #206 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/206/)
          . Allow fetching of delegation token from NameNode for hftp. Contributed by Jakob Homan.

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #206 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/206/ ) . Allow fetching of delegation token from NameNode for hftp. Contributed by Jakob Homan.
          Hide
          Jitendra Nath Pandey added a comment -

          Patch for hadoop 20.

          Show
          Jitendra Nath Pandey added a comment - Patch for hadoop 20.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12437295/HDFS-994-5.patch
          against trunk revision 916873.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/254/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/254/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/254/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/254/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12437295/HDFS-994-5.patch against trunk revision 916873. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/254/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/254/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/254/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/254/console This message is automatically generated.
          Hide
          Arun C Murthy added a comment -

          I just committed this, thanks Jakob!

          Show
          Arun C Murthy added a comment - I just committed this, thanks Jakob!
          Hide
          Boris Shkolnik added a comment -

          oops

          Show
          Boris Shkolnik added a comment - oops
          Hide
          Tsz Wo Nicholas Sze added a comment -

          +1 HDFS-994-5.patch looks good. Thanks, Jakob.

          Show
          Tsz Wo Nicholas Sze added a comment - +1 HDFS-994 -5.patch looks good. Thanks, Jakob.
          Hide
          Jakob Homan added a comment -

          submitting patch.

          Show
          Jakob Homan added a comment - submitting patch.
          Hide
          Jakob Homan added a comment -

          HDFS-991 caused this to go stale. Updated to use a configuration.

          Show
          Jakob Homan added a comment - HDFS-991 caused this to go stale. Updated to use a configuration.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12437289/HDFS-994-4.patch
          against trunk revision 916873.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 javac. The patch appears to cause tar ant target to fail.

          -1 findbugs. The patch appears to cause Findbugs to fail.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/253/testReport/
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/253/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/253/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12437289/HDFS-994-4.patch against trunk revision 916873. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The patch appears to cause tar ant target to fail. -1 findbugs. The patch appears to cause Findbugs to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/253/testReport/ Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/253/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/253/console This message is automatically generated.
          Hide
          Jakob Homan added a comment -

          Paging Hudson for new review.

          Show
          Jakob Homan added a comment - Paging Hudson for new review.
          Hide
          Jakob Homan added a comment -

          Thanks for the review Nicholas. Updated patch to include all suggestions.

          Show
          Jakob Homan added a comment - Thanks for the review Nicholas. Updated patch to include all suggestions.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          +1 patch looks good. Minor suggestions:

          • The InputStream in (or dis) in getDTfromRemote(..) is not closed.
          • You may use java.lang.Void instead of Object in the generic type of PrivilegedExceptionAction
            +      ugi.doAs(new PrivilegedExceptionAction<Object>() {
            
          • ugiFinal can be eliminated by
            +    final UserGroupInformation ugi;
            +    try {
            +      ugi = getUGI(req);
            +    } catch(IOException ioe) {
            
          Show
          Tsz Wo Nicholas Sze added a comment - +1 patch looks good. Minor suggestions: The InputStream in (or dis) in getDTfromRemote(..) is not closed. You may use java.lang.Void instead of Object in the generic type of PrivilegedExceptionAction + ugi.doAs( new PrivilegedExceptionAction< Object >() { ugiFinal can be eliminated by + final UserGroupInformation ugi; + try { + ugi = getUGI(req); + } catch (IOException ioe) {
          Hide
          Jakob Homan added a comment -

          Failed test is known-bad cactus download. Patch is ready for review.

          Show
          Jakob Homan added a comment - Failed test is known-bad cactus download. Patch is ready for review.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12437141/HDFS-994-3.patch
          against trunk revision 916534.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/249/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/249/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/249/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/249/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12437141/HDFS-994-3.patch against trunk revision 916534. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/249/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/249/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/249/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/249/console This message is automatically generated.
          Hide
          Jakob Homan added a comment -

          Submitting patch again, hopefully Hudson shows up?

          Show
          Jakob Homan added a comment - Submitting patch again, hopefully Hudson shows up?
          Hide
          Jakob Homan added a comment -

          Updated patch verified in Kerberos environment.

          Show
          Jakob Homan added a comment - Updated patch verified in Kerberos environment.
          Hide
          Jakob Homan added a comment -

          Updated patch now with test-passing goodness. Not sure where Hudson is, so ran full unit tests manually. All pass. Test-patch is good too.

          [exec] +1 overall.  
          [exec] 
          [exec]     +1 @author.  The patch does not contain any @author tags.
          [exec] 
          [exec]     +1 tests included.  The patch appears to include 2 new or modified tests.
          [exec] 
          [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
          [exec] 
          [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
          [exec] 
          [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
          [exec] 
          [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
          
          Show
          Jakob Homan added a comment - Updated patch now with test-passing goodness. Not sure where Hudson is, so ran full unit tests manually. All pass. Test-patch is good too. [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 2 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
          Hide
          Jakob Homan added a comment -

          submitting patch.

          Show
          Jakob Homan added a comment - submitting patch.
          Hide
          Jakob Homan added a comment -

          Patch for review. Manually tested in secure environment, which works fine except for the webservice interface, since we don't yet have kerberos-authed webinteraction. However, did fail in the correct way.

          Show
          Jakob Homan added a comment - Patch for review. Manually tested in secure environment, which works fine except for the webservice interface, since we don't yet have kerberos-authed webinteraction. However, did fail in the correct way.

            People

            • Assignee:
              Jakob Homan
              Reporter:
              Jakob Homan
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development