[SPARK-31812] Spark to support the auto cancelation of delegation token when an Application completes

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.4.5
• Component/s: Spark Submit
• Assignee: Unassigned
• Reporter: Mohammad Islam
• Created: 25/May/20 08:14
• Updated: 01/Jun/20 04:14

Description

Context :
YARN application provides a client API setCancelTokensWhenComplete to manage the  delegation token(DT) lifecycle. By default, YARN cancels the DT when App finishes. However, the user can override this NOT to cancel the DT after the App completes. In some instances, this is required to lessen the HDFS/KMS memory footprints by reducing the outstanding DTs.
MR and TEZ already allow that through client config  such  as mapreduce.job.complete.cancel.delegation.tokens and tez.cancel.delegation.tokens.on.completion respectively_._

Proposal :
Currently, Spark doesn't support it. However, we may need to manage the lifecycle of DT outside YARN/Spark framework.