Currently, hftp uses http to the Namenode's https port, which doesn't work.
Distcp job fails with hsftp when https is enabled in insecure cluster
Hftp should support both SPNEGO and KSSL
Replaced Kerberized SSL for image transfer and fsck with SPNEGO-based solution
Closed upon release of Hadoop-1.1.0.
Fix was committed by Owen on 10/1/2012.
The patch with the doAs patch looks ok to me.
Good catch. In most cases, it doesn't matter since the getDelegationToken is called by the constructor, but it will in general. Here's the delta patch to put the doAs back.
The patch removes the ugi.doAs wrapper in HftpFileSystem.getDelegationToken, is that intended? The ugi in the filesystem could be different from the current user and that would change the existing behavior.
I had to fix a couple of the test cases to reflect the new service name in the tokens.
Ok, I've tested this patch with:
For all three cases, I've tested
To work with SSL, I also needed to fix HDFS-3993.
Also, this patch needs to support KSSL anyway otherwise distcp'ing from a pre-SPNEGO clusters using Hftp will fail. Ie we'd have to force all secure users to upgrade to SPNEGO.
Hftp using http to the Namenode's https port doesn't work on trunk and branch-2 either, I've got a fix for that up on HDFS-3983.
Thanks Daryn. Will leave this with Owen.
To follow on with Owen's comments, we can't remove the secure port methods. They need to be conditionalized based on whether kssl or spnego is enabled.
h3461_20120925.patch: updates TestHftpFileSystem. Otherwise, it will fail.
Unfortunately, the HDFS-2617 patch that went in made the KSSL configurable instead of removing it and thus this patch needs to follow suit. I'll refactor the current patch to use ssl when hadoop.security.use-weak-http-crypto is set to true and security is enabled.
h3461_20120924.patch: Owen's patch. I only made it apply to branch-1.
This is the 1.1 branch with HDFS-2617 applied.
We test hftp pretty heavily and haven't seen this bug. Which branch has this issue?