Have you tested HsftpFileSystem too? Do we even support encrypting the transfer data if spnego is enabled?
No I didn't, but you make a good point. Thoughts on what to do here? Perhaps we should disallow use of Hsftp if SSL (either KSSL or cert-based) isn't enabled?
The addition of useKssl(conf) seems rather invasive in the sense that many callers have to be modified to specifically have knowledge of kssl.
It's really not all that many callers. Maybe we can cut down on a few, but it's going to go from like 10 to 7, or something.
A simple boolean complicates the ability to add new auth systems in the future.
I'm skeptical that we'll be adding more HTTP auth systems to branch-1, so I'm hesitant to build up a generic system for something that will only have two actual options.
SecurityUtil.openSecureHttpConnection swaps out the https scheme with http if kssl is not enabled. Negates a bunch of changes in HftpFileSystem and DelegationTokenFetcher.
The reason I didn't do this is because use of KSSL is only within HDFS, whereas SecurityUtil is in Common, and I didn't want to add a dependency on HDFS to Common. That said, it might be reasonable to move the dfs.use.kssl.auth key to Common, change it to "hadoop.security.use.kssl.auth", and then move the NameNode#useKsslAuth method to SecurityUtil itself. How does this sound?
Add a NameNode.getSecurePort(conf) that can use kssl to determine if the https or http port should be returned, HftpFileSystem could use this for the default secure port to be agnostic to kssl.
Seems like a good idea.
Maybe add an arg to the ctor of HttpServer for the auth filter, or add a setter for the auth filter so addInternalServlet and the many calls to it don't need to be modified.
I don't think this is reasonable since a single HttpServer might very well have individual servlets that have different auth filter requirements.
The initialization of a secure HttpServer in places such as the NN and 2NN seem virtually identical, maybe create a common method? Would centralize one of the main kssl checks.
I'll take a look at options for refactoring this, but I honestly don't think there's a lot of code that can be shared, or any new method that would be added would have to be heavily parameterized for just two call sites.
A few places appear to assume that if kssl is off, that the connection must be spnego w/o even checking if security is enabled.
Can you point them out specifically?