Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.24.0, 0.23.3, 2.0.0-alpha
    • Fix Version/s: 0.23.3, 2.0.2-alpha
    • Component/s: hdfs-client
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Hftp tries to select a token based on the non-secure port in the uri, instead of the secure-port. This breaks hftp on a secure cluster and there is no workaround.

        Issue Links

          Activity

          Hide
          Daryn Sharp added a comment -

          Token service changes erroneously started using the port in the uri.

          Show
          Daryn Sharp added a comment - Token service changes erroneously started using the port in the uri.
          Hide
          Daryn Sharp added a comment -

          Update hftp to use secure port to find tokens. Prior changes will select tokens based on uri, so updated hftp and hftps to store uris instead of socket addrs for the secure and non-secure uris.

          Show
          Daryn Sharp added a comment - Update hftp to use secure port to find tokens. Prior changes will select tokens based on uri, so updated hftp and hftps to store uris instead of socket addrs for the secure and non-secure uris.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12523807/HDFS-3312.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2313//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2313//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12523807/HDFS-3312.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2313//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2313//console This message is automatically generated.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          For hftp, the NN URI is non-secure but the dt needs to use secure URI.
          For hsftp, the NN URI is secure and the dt uses the same URI.

          +1 patch looks good.

          One minor suggestion for the future: we may use the scheme of nnUri in HftpFileSystem.getNamenodeURL(..). Then HsftpFileSystem.openConnection(..) could also use HftpFileSystem.getNamenodeURL(..).

          Show
          Tsz Wo Nicholas Sze added a comment - For hftp, the NN URI is non-secure but the dt needs to use secure URI. For hsftp, the NN URI is secure and the dt uses the same URI. +1 patch looks good. One minor suggestion for the future: we may use the scheme of nnUri in HftpFileSystem.getNamenodeURL(..). Then HsftpFileSystem.openConnection(..) could also use HftpFileSystem.getNamenodeURL(..).
          Hide
          Tsz Wo Nicholas Sze added a comment -

          I have committed this. Thanks, Daryn!

          Show
          Tsz Wo Nicholas Sze added a comment - I have committed this. Thanks, Daryn!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2194 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2194/)
          HDFS-3312. In HftpFileSystem, the namenode URI is non-secure but the delegation tokens have to use secure URI. Contributed by Daryn Sharp (Revision 1329462)

          Result = SUCCESS
          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1329462
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HftpFileSystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HsftpFileSystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestHftpDelegationToken.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2194 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2194/ ) HDFS-3312 . In HftpFileSystem, the namenode URI is non-secure but the delegation tokens have to use secure URI. Contributed by Daryn Sharp (Revision 1329462) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1329462 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HftpFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HsftpFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestHftpDelegationToken.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2120 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2120/)
          HDFS-3312. In HftpFileSystem, the namenode URI is non-secure but the delegation tokens have to use secure URI. Contributed by Daryn Sharp (Revision 1329462)

          Result = SUCCESS
          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1329462
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HftpFileSystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HsftpFileSystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestHftpDelegationToken.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2120 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2120/ ) HDFS-3312 . In HftpFileSystem, the namenode URI is non-secure but the delegation tokens have to use secure URI. Contributed by Daryn Sharp (Revision 1329462) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1329462 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HftpFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HsftpFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestHftpDelegationToken.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #2136 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2136/)
          HDFS-3312. In HftpFileSystem, the namenode URI is non-secure but the delegation tokens have to use secure URI. Contributed by Daryn Sharp (Revision 1329462)

          Result = SUCCESS
          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1329462
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HftpFileSystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HsftpFileSystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestHftpDelegationToken.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2136 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2136/ ) HDFS-3312 . In HftpFileSystem, the namenode URI is non-secure but the delegation tokens have to use secure URI. Contributed by Daryn Sharp (Revision 1329462) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1329462 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HftpFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HsftpFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestHftpDelegationToken.java
          Hide
          Daryn Sharp added a comment -

          One minor suggestion for the future: we may use the scheme of nnUri in HftpFileSystem.getNamenodeURL(..). Then HsftpFileSystem.openConnection(..) could also use HftpFileSystem.getNamenodeURL(..).

          I noticed that too. Hsftp isn't using the secure port for data transfers. I already made a patch and test for it, but when I tried to run the hftp tests with hsftp it produced some bizarre exceptions that I haven't had a chance to debug.

          Show
          Daryn Sharp added a comment - One minor suggestion for the future: we may use the scheme of nnUri in HftpFileSystem.getNamenodeURL(..). Then HsftpFileSystem.openConnection(..) could also use HftpFileSystem.getNamenodeURL(..). I noticed that too. Hsftp isn't using the secure port for data transfers. I already made a patch and test for it, but when I tried to run the hftp tests with hsftp it produced some bizarre exceptions that I haven't had a chance to debug.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #237 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/237/)
          svn merge -c 1329462 from trunk for HDFS-3312. In HftpFileSystem, the namenode URI is non-secure but the delegation tokens have to use secure URI. (Revision 1329465)

          Result = SUCCESS
          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1329465
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HftpFileSystem.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HsftpFileSystem.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestHftpDelegationToken.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #237 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/237/ ) svn merge -c 1329462 from trunk for HDFS-3312 . In HftpFileSystem, the namenode URI is non-secure but the delegation tokens have to use secure URI. (Revision 1329465) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1329465 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HftpFileSystem.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HsftpFileSystem.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestHftpDelegationToken.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1024 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1024/)
          HDFS-3312. In HftpFileSystem, the namenode URI is non-secure but the delegation tokens have to use secure URI. Contributed by Daryn Sharp (Revision 1329462)

          Result = FAILURE
          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1329462
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HftpFileSystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HsftpFileSystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestHftpDelegationToken.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1024 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1024/ ) HDFS-3312 . In HftpFileSystem, the namenode URI is non-secure but the delegation tokens have to use secure URI. Contributed by Daryn Sharp (Revision 1329462) Result = FAILURE szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1329462 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HftpFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HsftpFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestHftpDelegationToken.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1059 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1059/)
          HDFS-3312. In HftpFileSystem, the namenode URI is non-secure but the delegation tokens have to use secure URI. Contributed by Daryn Sharp (Revision 1329462)

          Result = FAILURE
          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1329462
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HftpFileSystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HsftpFileSystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestHftpDelegationToken.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1059 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1059/ ) HDFS-3312 . In HftpFileSystem, the namenode URI is non-secure but the delegation tokens have to use secure URI. Contributed by Daryn Sharp (Revision 1329462) Result = FAILURE szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1329462 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HftpFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HsftpFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestHftpDelegationToken.java

            People

            • Assignee:
              Daryn Sharp
              Reporter:
              Daryn Sharp
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development