[SPARK-25689] Move token renewal logic to driver in yarn-client mode

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 3.0.0
• Component/s: Spark Core, YARN
• Labels: pull-request-available
• Assignee: Marcelo Masiero Vanzin
• Reporter: Marcelo Masiero Vanzin
• Created: 09/Oct/18 18:39
• Updated: 29/Dec/23 00:15
• Resolved: 07/Jan/19 20:40

Description

Currently, both in yarn-cluster and yarn-client mode, the YARN AM is responsible for renewing delegation tokens. That differs from other RMs (Mesos and later k8s when it supports this functionality), and is one of the roadblocks towards fully sharing the same delegation token-related code.

We should look at keeping the renewal logic within the driver in yarn-client mode. That would also remove the need to distribute the user's keytab to the AM when running in that particular mode.

Issue Links

• relates to: SPARK-14743 Improve delegation token handling in secure clusters
• links to: GitHub Pull Request #23338
• links to: GitHub Pull Request #23689
• links to: GitHub Pull Request #23922
• links to: GitHub Pull Request #42890