Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.20.205.0
    • Fix Version/s: 0.20.205.0, 0.23.0
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Distcp with hftp is failing.

      $hadoop   distcp hftp://<NNhostname>:50070/user/hadoopqa/1316814737/newtemp 1316814737/as
      11/09/23 21:52:33 INFO tools.DistCp: srcPaths=[hftp://<NNhostname>:50070/user/hadoopqa/1316814737/newtemp]
      11/09/23 21:52:33 INFO tools.DistCp: destPath=1316814737/as
      Retrieving token from: https://<NN IP>:50470/getDelegationToken
      Retrieving token from: https://<NN IP>:50470/getDelegationToken?renewer=mapred
      11/09/23 21:52:34 INFO security.TokenCache: Got dt for hftp://<NNhostname>:50070/user/hadoopqa/1316814737/newtemp;uri=<NN IP>:50470;t.service=<NN IP>:50470
      org.apache.hadoop.ipc.RemoteException: name=hadoopqa != expected=null
      	at org.apache.hadoop.ipc.RemoteException.valueOf(RemoteException.java:102)
      	at org.apache.hadoop.hdfs.HftpFileSystem$LsParser.startElement(HftpFileSystem.java:385)
      	...
      
      1. HDFS-2361.trunk.1.patch
        3 kB
        Jitendra Nath Pandey
      2. HDFS-2361.trunk.2.patch
        3 kB
        Jitendra Nath Pandey
      3. HDFS-2361-20s.1.patch
        1 kB
        Jitendra Nath Pandey
      4. HDFS-2361-20s.2.patch
        1 kB
        Jitendra Nath Pandey
      5. HDFS-2361-20s.3.patch
        2 kB
        Jitendra Nath Pandey

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Patch Available Patch Available Open Open
          5h 12m 1 Jitendra Nath Pandey 27/Sep/11 09:28
          Open Open Patch Available Patch Available
          3d 5h 18m 2 Jitendra Nath Pandey 27/Sep/11 09:28
          Patch Available Patch Available Resolved Resolved
          21h 3m 1 Jitendra Nath Pandey 28/Sep/11 06:31
          Resolved Resolved Closed Closed
          20d 18h 54m 1 Matt Foley 19/Oct/11 01:26
          Allen Wittenauer made changes -
          Fix Version/s 2.0.0-alpha [ 12320353 ]
          Allen Wittenauer made changes -
          Fix Version/s 2.0.0-alpha [ 12320353 ]
          Fix Version/s 0.24.0 [ 12317653 ]
          Gavin made changes -
          Link This issue is depended upon by HDFS-2369 [ HDFS-2369 ]
          Gavin made changes -
          Link This issue blocks HDFS-2369 [ HDFS-2369 ]
          Suresh Srinivas made changes -
          Fix Version/s 1.1.0 [ 12317959 ]
          Tsz Wo Nicholas Sze made changes -
          Link This issue is related to HDFS-2316 [ HDFS-2316 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-0.23-Build #64 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/64/)
          svn merge -c 1176729 from trunk for HDFS-2361.

          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189440
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/JspHelper.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #64 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/64/ ) svn merge -c 1176729 from trunk for HDFS-2361 . szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189440 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/JspHelper.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #52 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/52/)
          svn merge -c 1176729 from trunk for HDFS-2361.

          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189440
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/JspHelper.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #52 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/52/ ) svn merge -c 1176729 from trunk for HDFS-2361 . szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189440 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/JspHelper.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-0.23-Commit #58 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/58/)
          svn merge -c 1176729 from trunk for HDFS-2361.

          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189440
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/JspHelper.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Commit #58 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/58/ ) svn merge -c 1176729 from trunk for HDFS-2361 . szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189440 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/JspHelper.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Commit #63 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/63/)
          svn merge -c 1176729 from trunk for HDFS-2361.

          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189440
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/JspHelper.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Commit #63 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/63/ ) svn merge -c 1176729 from trunk for HDFS-2361 . szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189440 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/JspHelper.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-0.23-Commit #62 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/62/)
          svn merge -c 1176729 from trunk for HDFS-2361.

          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189440
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/JspHelper.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-0.23-Commit #62 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/62/ ) svn merge -c 1176729 from trunk for HDFS-2361 . szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189440 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/JspHelper.java
          Tsz Wo Nicholas Sze made changes -
          Fix Version/s 0.23.0 [ 12315571 ]
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Merged to 0.23.

          Show
          Tsz Wo Nicholas Sze added a comment - Merged to 0.23.
          Matt Foley made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Hide
          Matt Foley added a comment -

          Closed upon release of 0.20.205.0

          Show
          Matt Foley added a comment - Closed upon release of 0.20.205.0
          Daryn Sharp made changes -
          Link This issue relates to HDFS-2380 [ HDFS-2380 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #844 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/844/)
          HDFS-2361. hftp is broken, fixed username checks in JspHelper.

          jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1176729
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/JspHelper.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #844 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/844/ ) HDFS-2361 . hftp is broken, fixed username checks in JspHelper. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1176729 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/JspHelper.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #814 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/814/)
          HDFS-2361. hftp is broken, fixed username checks in JspHelper.

          jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1176729
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/JspHelper.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #814 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/814/ ) HDFS-2361 . hftp is broken, fixed username checks in JspHelper. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1176729 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/JspHelper.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #995 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/995/)
          HDFS-2361. hftp is broken, fixed username checks in JspHelper.

          jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1176729
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/JspHelper.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #995 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/995/ ) HDFS-2361 . hftp is broken, fixed username checks in JspHelper. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1176729 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/JspHelper.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #974 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/974/)
          HDFS-2361. hftp is broken, fixed username checks in JspHelper.

          jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1176729
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/JspHelper.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #974 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/974/ ) HDFS-2361 . hftp is broken, fixed username checks in JspHelper. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1176729 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/JspHelper.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #1052 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1052/)
          HDFS-2361. hftp is broken, fixed username checks in JspHelper.

          jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1176729
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/JspHelper.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1052 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1052/ ) HDFS-2361 . hftp is broken, fixed username checks in JspHelper. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1176729 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/JspHelper.java
          Jitendra Nath Pandey made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Fix Version/s 0.20.205.0 [ 12316392 ]
          Fix Version/s 0.20.206.0 [ 12317959 ]
          Fix Version/s 0.24.0 [ 12317653 ]
          Resolution Fixed [ 1 ]
          Hide
          Jitendra Nath Pandey added a comment -

          This is committed.

          Show
          Jitendra Nath Pandey added a comment - This is committed.
          Hide
          Jitendra Nath Pandey added a comment -

          The patch was manually tested on a secure cluster in branch-0.20-security.
          This issue showed up only in a secure cluster, and the code change in the patch takes effect only if security is enabled. Therefore, tests require a secure setup.

          Show
          Jitendra Nath Pandey added a comment - The patch was manually tested on a secure cluster in branch-0.20-security. This issue showed up only in a secure cluster, and the code change in the patch takes effect only if security is enabled. Therefore, tests require a secure setup.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          +1 on HDFS-2361.trunk.2.patch

          Show
          Tsz Wo Nicholas Sze added a comment - +1 on HDFS-2361 .trunk.2.patch
          Tsz Wo Nicholas Sze made changes -
          Description Distcp with hftp is failing.


          $hadoop distcp hftp://&lt;NNhostname&gt;:50070/user/hadoopqa/1316814737/newtemp 1316814737/as
          11/09/23 21:52:33 INFO tools.DistCp: srcPaths=[hftp://&lt;NNhostname&gt;:50070/user/hadoopqa/1316814737/newtemp]
          11/09/23 21:52:33 INFO tools.DistCp: destPath=1316814737/as
          Retrieving token from: https://&lt;NN IP>:50470/getDelegationToken
          Retrieving token from: https://&lt;NN IP>:50470/getDelegationToken?renewer=mapred
          11/09/23 21:52:34 INFO security.TokenCache: Got dt for hftp://&lt;NNhostname&gt;:50070/user/hadoopqa/1316814737/newtemp;uri=&lt;NN IP>:50470;t.service=<NN IP>:50470
          org.apache.hadoop.ipc.RemoteException: name=hadoopqa != expected=null
          at org.apache.hadoop.ipc.RemoteException.valueOf(RemoteException.java:102)
          at org.apache.hadoop.hdfs.HftpFileSystem$LsParser.startElement(HftpFileSystem.java:385)
          at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.startElement(AbstractSAXParser.java:501)
          at com.sun.org.apache.xerces.internal.parsers.AbstractXMLDocumentParser.emptyElement(AbstractXMLDocumentParser.java:179)
          at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.scanStartElement(XMLNSDocumentScannerImpl.java:377)
          at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl$NSContentDriver.scanRootElementHook(XMLNSDocumentScannerImpl.java:626)
          at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:3103)
          at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:922)
          at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:648)
          at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:140)
          at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:511)
          at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:808)
          at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:737)
          at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:119)
          at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1205)
          at org.apache.hadoop.hdfs.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:426)
          at org.apache.hadoop.hdfs.HftpFileSystem$LsParser.getFileStatus(HftpFileSystem.java:437)
          at org.apache.hadoop.hdfs.HftpFileSystem.getFileStatus(HftpFileSystem.java:466)
          at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:708)
          at org.apache.hadoop.tools.DistCp.checkSrcPath(DistCp.java:636)
          at org.apache.hadoop.tools.DistCp.copy(DistCp.java:656)
          at org.apache.hadoop.tools.DistCp.run(DistCp.java:881)
          at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
          at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
          at org.apache.hadoop.tools.DistCp.main(DistCp.java:908)
          Distcp with hftp is failing.

          {noformat}
          $hadoop distcp hftp://&lt;NNhostname&gt;:50070/user/hadoopqa/1316814737/newtemp 1316814737/as
          11/09/23 21:52:33 INFO tools.DistCp: srcPaths=[hftp://&lt;NNhostname&gt;:50070/user/hadoopqa/1316814737/newtemp]
          11/09/23 21:52:33 INFO tools.DistCp: destPath=1316814737/as
          Retrieving token from: https://&lt;NN IP>:50470/getDelegationToken
          Retrieving token from: https://&lt;NN IP>:50470/getDelegationToken?renewer=mapred
          11/09/23 21:52:34 INFO security.TokenCache: Got dt for hftp://&lt;NNhostname&gt;:50070/user/hadoopqa/1316814737/newtemp;uri=&lt;NN IP>:50470;t.service=<NN IP>:50470
          org.apache.hadoop.ipc.RemoteException: name=hadoopqa != expected=null
          at org.apache.hadoop.ipc.RemoteException.valueOf(RemoteException.java:102)
          at org.apache.hadoop.hdfs.HftpFileSystem$LsParser.startElement(HftpFileSystem.java:385)
          ...
          {noformat}
          Hide
          Jitendra Nath Pandey added a comment -

          > It is not a spnego implementation.
          Correction: It is not a spnego limitation.

          That said, it can be debated (and has been debated before) whether we want to switch to long names consistently in hadoop. But that requires revisiting, all the contexts where we assumed shortnames were sufficient.

          Show
          Jitendra Nath Pandey added a comment - > It is not a spnego implementation. Correction: It is not a spnego limitation. That said, it can be debated (and has been debated before) whether we want to switch to long names consistently in hadoop. But that requires revisiting, all the contexts where we assumed shortnames were sufficient.
          Hide
          Jitendra Nath Pandey added a comment -

          It is not a spnego implementation. KerberosAuthenticationHandler is one example where we use shortnames. There are other places as well in hadoop, where it was assumed that it is sufficient to use shortnames, for example proxy user configuration, in checking file permissions etc.

          Show
          Jitendra Nath Pandey added a comment - It is not a spnego implementation. KerberosAuthenticationHandler is one example where we use shortnames. There are other places as well in hadoop, where it was assumed that it is sufficient to use shortnames, for example proxy user configuration, in checking file permissions etc.
          Hide
          Daryn Sharp added a comment -

          Would it be better to change this to not use getShortName()?

          KerberosAuthenticationHandler
          KerberosName kerberosName = new KerberosName(clientPrincipal);
          String userName = kerberosName.getShortName();
          token = new AuthenticationToken(userName, clientPrincipal, TYPE);
          

          There is a comment in the code: "The realm can be omitted from the principal as the JDK GSS libraries will use the realm name of the configured default realm". It would appear that this jira may be due to an arbitrary, and maybe unnecessary, implementation decision?

          Show
          Daryn Sharp added a comment - Would it be better to change this to not use getShortName() ? KerberosAuthenticationHandler KerberosName kerberosName = new KerberosName(clientPrincipal); String userName = kerberosName.getShortName(); token = new AuthenticationToken(userName, clientPrincipal, TYPE); There is a comment in the code: "The realm can be omitted from the principal as the JDK GSS libraries will use the realm name of the configured default realm". It would appear that this jira may be due to an arbitrary, and maybe unnecessary, implementation decision?
          Hide
          Daryn Sharp added a comment -

          Does this weaken the security of hadoop? Are there other safeguards to prevent a user in one realm from accessing the files of an identically named user in a different realm? If not, is it correct that multiple-realm installations now have a vulnerability due to what I believe you are saying is a spnego limitation?

          Show
          Daryn Sharp added a comment - Does this weaken the security of hadoop? Are there other safeguards to prevent a user in one realm from accessing the files of an identically named user in a different realm? If not, is it correct that multiple-realm installations now have a vulnerability due to what I believe you are saying is a spnego limitation?
          Hide
          Jitendra Nath Pandey added a comment -

          That's correct. It is assumed that short-names will be unique across users.
          KerberosAuthenticationHandler sets only the short name in the request, therefore only short name is available in JspHelper#getUGI in case of kerberos spnego authentication.

          Show
          Jitendra Nath Pandey added a comment - That's correct. It is assumed that short-names will be unique across users. KerberosAuthenticationHandler sets only the short name in the request, therefore only short name is available in JspHelper#getUGI in case of kerberos spnego authentication.
          Hide
          Daryn Sharp added a comment -

          Just a question: since only the short names are being compared, does this mean that same user in two different realms are considered to be equivalent?

          Show
          Daryn Sharp added a comment - Just a question: since only the short names are being compared, does this mean that same user in two different realms are considered to be equivalent?
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12496649/HDFS-2361.trunk.2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hdfs.TestDfsOverAvroRpc
          org.apache.hadoop.hdfs.server.blockmanagement.TestHost2NodesMap
          org.apache.hadoop.hdfs.server.datanode.TestReplicasMap

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1297//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1297//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12496649/HDFS-2361.trunk.2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.TestDfsOverAvroRpc org.apache.hadoop.hdfs.server.blockmanagement.TestHost2NodesMap org.apache.hadoop.hdfs.server.datanode.TestReplicasMap +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1297//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1297//console This message is automatically generated.
          Jitendra Nath Pandey made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Jitendra Nath Pandey made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Jitendra Nath Pandey made changes -
          Attachment HDFS-2361.trunk.2.patch [ 12496649 ]
          Hide
          Jitendra Nath Pandey added a comment -

          Uploaded wrong patch earlier. This one fixes the import.

          Show
          Jitendra Nath Pandey added a comment - Uploaded wrong patch earlier. This one fixes the import.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          In trunk, KerberosName is in org.apache.hadoop.security.authentication.util. So the import incorrect.

          Show
          Tsz Wo Nicholas Sze added a comment - In trunk, KerberosName is in org.apache.hadoop.security.authentication.util. So the import incorrect.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12496624/HDFS-2361.trunk.1.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 javac. The patch appears to cause tar ant target to fail.

          -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1295//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1295//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12496624/HDFS-2361.trunk.1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The patch appears to cause tar ant target to fail. -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1295//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1295//console This message is automatically generated.
          Tsz Wo Nicholas Sze made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hadoop Flags Reviewed [ 10343 ]
          Hide
          Tsz Wo Nicholas Sze added a comment -

          +1 the patch for trunk looks good.

          Show
          Tsz Wo Nicholas Sze added a comment - +1 the patch for trunk looks good.
          Hide
          Jitendra Nath Pandey added a comment -

          Committed to 205,206.

          Show
          Jitendra Nath Pandey added a comment - Committed to 205,206.
          Jitendra Nath Pandey made changes -
          Link This issue blocks HDFS-2369 [ HDFS-2369 ]
          Jitendra Nath Pandey made changes -
          Attachment HDFS-2361.trunk.1.patch [ 12496624 ]
          Hide
          Jitendra Nath Pandey added a comment -

          The patch for trunk is attached.

          Show
          Jitendra Nath Pandey added a comment - The patch for trunk is attached.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          +1 the new patch looks good.

          Show
          Tsz Wo Nicholas Sze added a comment - +1 the new patch looks good.
          Jitendra Nath Pandey made changes -
          Attachment HDFS-2361-20s.3.patch [ 12496619 ]
          Hide
          Jitendra Nath Pandey added a comment -

          Uploaded a new patch, because short names should be compared.

          Show
          Jitendra Nath Pandey added a comment - Uploaded a new patch, because short names should be compared.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          +1 the 0.20s patch looks good. Thanks for fixing it.

          Please also post a patch for trunk.

          Show
          Tsz Wo Nicholas Sze added a comment - +1 the 0.20s patch looks good. Thanks for fixing it. Please also post a patch for trunk.
          Jitendra Nath Pandey made changes -
          Assignee Jitendra Nath Pandey [ jnp ]
          Jitendra Nath Pandey made changes -
          Attachment HDFS-2361-20s.2.patch [ 12496593 ]
          Hide
          Jitendra Nath Pandey added a comment -

          Updated patch. If token string is present, the username from query and the 'remote user' should be matched to the user in token.

          Show
          Jitendra Nath Pandey added a comment - Updated patch. If token string is present, the username from query and the 'remote user' should be matched to the user in token.
          Hide
          Jitendra Nath Pandey added a comment -

          Uploaded a patch.

          Show
          Jitendra Nath Pandey added a comment - Uploaded a patch.
          Jitendra Nath Pandey made changes -
          Attachment HDFS-2361-20s.1.patch [ 12496592 ]
          Rajit Saha made changes -
          Summary distcp with hftp is failing hftp is broken
          Rajit Saha made changes -
          Release Note It seems hftp itself is broken


          $hadoop dfs -lsr hftp://&lt;NN hostname>:50070/tmp
          Retrieving token from: https://&lt;NN IP>:50470/getDelegationToken
          lsr: name=hadoopqa != expected=null
          Hide
          Rajit Saha added a comment -

          It seems hftp itself is broken

          $hadoop dfs -lsr hftp://<NN hostname>:50070/tmp
          Retrieving token from: https://<NN IP>:50470/getDelegationToken
          lsr: name=hadoopqa != expected=null

          Show
          Rajit Saha added a comment - It seems hftp itself is broken $hadoop dfs -lsr hftp://<NN hostname>:50070/tmp Retrieving token from: https://<NN IP>:50470/getDelegationToken lsr: name=hadoopqa != expected=null
          Rajit Saha made changes -
          Field Original Value New Value
          Release Note It seems hftp itself is broken


          $hadoop dfs -lsr hftp://&lt;NN hostname>:50070/tmp
          Retrieving token from: https://&lt;NN IP>:50470/getDelegationToken
          lsr: name=hadoopqa != expected=null
          Priority Major [ 3 ] Critical [ 2 ]
          Rajit Saha created issue -

            People

            • Assignee:
              Jitendra Nath Pandey
              Reporter:
              Rajit Saha
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development