[SPARK-37205] Support mapreduce.job.send-token-conf when starting containers in YARN

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.3.0
• Component/s: YARN
• Assignee: Chao Sun
• Reporter: Chao Sun
• Created: 03/Nov/21 23:45
• Updated: 01/Jul/22 16:54
• Resolved: 08/Dec/21 17:25

Description

mapreduce.job.send-token-conf is a useful feature in Hadoop (see YARN-5910 with which RM is not required to statically have config for all the secure HDFS clusters. Currently it only works for MRv2 but it'd be nice if Spark can also use this feature. I think we only need to pass the config to LaunchContainerContext in Client.createContainerLaunchContext.

Issue Links

• causes: SPARK-39657 YARN AM client should call the non-static setTokensConf method
• is related to: YARN-5910 Support for multi-cluster delegation tokens
• links to: [Github] Pull Request #34635 (sunchao)
• links to: [Github] Pull Request #37037 (sunchao)
• links to: [Github] Pull Request #37037 (sunchao)