Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-25689

Move token renewal logic to driver in yarn-client mode

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.4.0
    • Fix Version/s: 3.0.0
    • Component/s: Spark Core, YARN
    • Labels:
      None

      Description

      Currently, both in yarn-cluster and yarn-client mode, the YARN AM is responsible for renewing delegation tokens. That differs from other RMs (Mesos and later k8s when it supports this functionality), and is one of the roadblocks towards fully sharing the same delegation token-related code.

      We should look at keeping the renewal logic within the driver in yarn-client mode. That would also remove the need to distribute the user's keytab to the AM when running in that particular mode.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                vanzin Marcelo Masiero Vanzin
                Reporter:
                vanzin Marcelo Masiero Vanzin
              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: