Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1007

HFTP needs to be updated to use delegation tokens

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.22.0
    • Component/s: security
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      HFTPFileSystem should be updated to use the delegation tokens so that it can talk to the secure namenodes.

      1. hdfs-1007-securityutil-fix.patch
        0.9 kB
        Devaraj Das
      2. hdfs-1007-long-running-hftp-client.patch
        57 kB
        Devaraj Das
      3. HDFS-1007-BP20-fix-3.patch
        31 kB
        Boris Shkolnik
      4. HDFS-1007-BP20-fix-2.patch
        30 kB
        Boris Shkolnik
      5. HDFS-1007-BP20-fix-1.patch
        12 kB
        Boris Shkolnik
      6. HDFS-1007-BP20.patch
        3 kB
        Boris Shkolnik
      7. HDFS-1007-3.patch
        38 kB
        Boris Shkolnik
      8. HDFS-1007-2.patch
        38 kB
        Boris Shkolnik
      9. HDFS-1007-1.patch
        38 kB
        Boris Shkolnik
      10. distcp-hftp-2.1.1.patch
        3 kB
        Devaraj Das
      11. distcp-hftp.patch
        18 kB
        Devaraj Das
      12. distcp-hftp.2.patch
        19 kB
        Devaraj Das
      13. distcp-hftp.2.1.patch
        1 kB
        Devaraj Das
      14. distcp-hftp.1.patch
        18 kB
        Devaraj Das
      15. 1007-bugfix.patch
        2 kB
        Devaraj Das

        Issue Links

          Activity

          Hide
          Devaraj Das added a comment -

          Patch for Y20. Not for commit.

          Show
          Devaraj Das added a comment - Patch for Y20. Not for commit.
          Hide
          Devaraj Das added a comment -

          Some minor changes.

          Show
          Devaraj Das added a comment - Some minor changes.
          Hide
          Devaraj Das added a comment -

          Updates HsFtpFileSystem.. Patch for Y20. Not for commit here.

          Show
          Devaraj Das added a comment - Updates HsFtpFileSystem.. Patch for Y20. Not for commit here.
          Hide
          Devaraj Das added a comment -

          This patch is a bugfix on top of the distcp-hftp.2.patch.

          Show
          Devaraj Das added a comment - This patch is a bugfix on top of the distcp-hftp.2.patch.
          Hide
          Kan Zhang added a comment -

          The delegation token should be fetched by distcp client, not by HftpFilesystem (or HsftpFilesystem).

          Show
          Kan Zhang added a comment - The delegation token should be fetched by distcp client, not by HftpFilesystem (or HsftpFilesystem).
          Hide
          Boris Shkolnik added a comment -

          ugi over http fix. for previous version, not for commit

          Show
          Boris Shkolnik added a comment - ugi over http fix. for previous version, not for commit
          Hide
          Devaraj Das added a comment -

          Patch adding a bugfix in the last patch that Boris attached.

          Show
          Devaraj Das added a comment - Patch adding a bugfix in the last patch that Boris attached.
          Hide
          Boris Shkolnik added a comment -

          for prev version, not for Commit

          Show
          Boris Shkolnik added a comment - for prev version, not for Commit
          Hide
          Boris Shkolnik added a comment -

          for prev. version. fix for the patch.

          Show
          Boris Shkolnik added a comment - for prev. version. fix for the patch.
          Hide
          Devaraj Das added a comment -

          Bugfix for handling null tokens (for Y20S)

          Show
          Devaraj Das added a comment - Bugfix for handling null tokens (for Y20S)
          Hide
          Devaraj Das added a comment -

          Attaching a patch that makes long running servers using hftp work. Also has some refactoring in the MR code to do with handling of delegation tokens. Patch for Y!20S.

          Show
          Devaraj Das added a comment - Attaching a patch that makes long running servers using hftp work. Also has some refactoring in the MR code to do with handling of delegation tokens. Patch for Y!20S.
          Hide
          Devaraj Das added a comment -

          Attaching a patch that fixes a problem in SecurityUtil.buildDTServiceName to do with handling of null host in the URI. Patch for Y20S.

          Show
          Devaraj Das added a comment - Attaching a patch that fixes a problem in SecurityUtil.buildDTServiceName to do with handling of null host in the URI. Patch for Y20S.
          Hide
          Boris Shkolnik added a comment -

          patch for trunk
          includes:
          1007-bugfix.patch
          HDFS-1007-BP20-fix-3.patch
          distcp-hftp-2.1.1.patch
          distcp-hftp.2.1.patch
          distcp-hftp.2.patch

          Show
          Boris Shkolnik added a comment - patch for trunk includes: 1007-bugfix.patch HDFS-1007 -BP20-fix-3.patch distcp-hftp-2.1.1.patch distcp-hftp.2.1.patch distcp-hftp.2.patch
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12449130/HDFS-1007-1.patch
          against trunk revision 962696.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 javadoc. The javadoc tool appears to have generated 1 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 1 new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/429/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/429/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/429/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/429/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12449130/HDFS-1007-1.patch against trunk revision 962696. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc. The javadoc tool appears to have generated 1 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/429/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/429/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/429/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/429/console This message is automatically generated.
          Hide
          Boris Shkolnik added a comment -

          fixed javadoc and findbug warnings
          Ran all the tests - all passed (except TestHdfsTrash).

          There are not tests in this patch. It will be tested by the test in TestTokenCache (in MR).

          Show
          Boris Shkolnik added a comment - fixed javadoc and findbug warnings Ran all the tests - all passed (except TestHdfsTrash). There are not tests in this patch. It will be tested by the test in TestTokenCache (in MR).
          Hide
          Devaraj Das added a comment -

          Looks fine. Is the MR patch going to test hftp end-to-end (via distcp) ?

          Show
          Devaraj Das added a comment - Looks fine. Is the MR patch going to test hftp end-to-end (via distcp) ?
          Hide
          Boris Shkolnik added a comment -

          TestTokenCache will test getting delegation token over HTTP...

          Show
          Boris Shkolnik added a comment - TestTokenCache will test getting delegation token over HTTP...
          Hide
          Boris Shkolnik added a comment -

          merged with trunk

          Show
          Boris Shkolnik added a comment - merged with trunk
          Hide
          Boris Shkolnik added a comment -

          committed to trunk

          Show
          Boris Shkolnik added a comment - committed to trunk
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #342 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/342/)
          HDFS-1007. HFTP needs to be updated to use delegation tokens

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #342 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/342/ ) HDFS-1007 . HFTP needs to be updated to use delegation tokens
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #370 (See https://hudson.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/370/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #370 (See https://hudson.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/370/ )

            People

            • Assignee:
              Devaraj Das
              Reporter:
              Devaraj Das
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development