Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-31812

Spark to support the auto cancelation of delegation token when an Application completes

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.4.5
    • Fix Version/s: None
    • Component/s: Spark Submit
    • Labels:
      None

      Description

      Context :
      YARN application provides a client API setCancelTokensWhenComplete to manage the  delegation token(DT) lifecycle. By default, YARN cancels the DT when App finishes. However, the user can override this NOT to cancel the DT after the App completes. In some instances, this is required to lessen the HDFS/KMS memory footprints by reducing the outstanding DTs.
      MR and TEZ already allow that through client config  such  as mapreduce.job.complete.cancel.delegation.tokens and tez.cancel.delegation.tokens.on.completion respectively_._

      Proposal :
      Currently, Spark doesn't support it. However, we may need to manage the lifecycle of DT outside YARN/Spark framework.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              kamrul Mohammad Islam
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: