Thanks Steve !Great material !
Some questions/comments I have
It is the responsibility of the application to renew all tokens other than the AMRM and timeline tokens.
I personally feel here the 'renew' word is a bit confusing. Two kinds of 'renew' we have. 1) Before tokens' final expiration, tokens submitted via applicaionSubmissionContext are automatically renewed by DelegationTokenRenwer in RM. 2) After the token final expiration, application has to re-renew(or 're-fetch') the token by themselves.
Should we clarify these two?
The AM must implement an IPC interface which permits containers to request a new set of delegation tokens; this interface must itself use authentication and ideally wire encryption.
Wonder how this works. Since container does not have keytab, so no kerberos channel. What kind of authentication is this to get the delegation tokens ?
Before a delegation token is due to expire, the processes running in the containers must request new tokens from the Application Master, over the IPC channel.
Not clear to me how this works. Say, if container wants to get a new hdfs delegation token, how does it get the new hdfs delegation token from AM? Is it because AM gets a new hdfs delegation token in the first place which then passed to container when container asks for it?
Because the RM refreshes tokens on AM restart,
Correct me if I'm wrong, RM will not refresh any delegation tokens on AM restart. It'll refresh AMRM token for sure.
A thread or executor is started to renew threads on a regular basis.
should it be "is started to renew 'tokens' " ?