Daryn Sharp, thanks for the detailed review. Attaching patch integrating your feedback as follows:
On #1, you right, i’ll remove the hash() and equals() methods from both, AuthenticatedURL.Token and DelegationTokenAuthenticatedURL.Token().
On #2, HttpFS remains on the context UGI, a doAs() is used to create the FS for the corresponding FS operation. This is not changing with this patch.
HADOOP-10836 (patch uploaded, coming on top of this) takes care of cleaning up this.
On #3, yes, you are right, here I’m taking care of typical HTTP HA (ie behind a VIP, no URL doctoring). We’ll need to add a hook for WebHdfs HA when doing WebHdfs HA.
On #4, For Authenticator to support token ops we would have to modify Authenticator in an incompatible way. That is why I’ve introduced the DelegationToken* versions of things both on server and client side. An Authenticator can work with an endpoint supporting delegation tokens, a DelegationTokenAuthenticator can work with an endpoint not supporting delegation tokens, the lowest denominator behavior (prior to embedded DT support) happen. With the exception that the lowest denominator implements in app support for delegation tokens.
On #5, proxyuser support is being added in follow up
HADOOP-10835 (patch uploaded), I believe there is happening what you are asking for.
On #6, the SC_BAD_REQUEST means that the request was syntactically incorrect, missing token param or wrong HTTP method for the token operation. I’m changing from SC_BAD_REQUEST to SC_NOT_FOUND when trying to cancel a delegation token not known to the token manager.
On #7, as with RPC, getDelegationToken() and renewDelegationToken() require Kerberos, and cancelDelegationToken() can be done with just a delegation token (it can be the same or other owned by the renewer).
On #8, the ephemeral token means that no hadoop-auth cookie is issue when a token a is presented, nothing else.
On #9, I guess we could improve this by adding an HTTP error code to the AuthenticationException, thoughts?
On 'However directly invoking the secret manager will short-circuit …', actually you can set you own token secret manager, if the webapp context is seeded with a token manager using the DelegationTokenAuthenticationFilter.DELEGATION_TOKEN_SECRET_MANAGER_ATTR, the DelegationTokenManager will use that instance instead creating one. This allows to wire the the NN secret manager.
On 'How ill it know to throw RetriableException…?', currently the server side exception is not propagated to the client for reconstruction in these classes. HttpFS does it, and also KMS does it. I’ll open a follow up JIRA do bring that here. Sounds good?