[SPARK-34295] Allow option similar to mapreduce.job.hdfs-servers.token-renewal.exclude

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.2.0
• Component/s: YARN
• Assignee: L. C. Hsieh
• Reporter: L. C. Hsieh
• Created: 29/Jan/21 23:31
• Updated: 24/Mar/21 08:12
• Resolved: 24/Mar/21 08:12

Description

MapReduce jobs can instruct YARN to skip renewal of tokens obtained from certain hosts by specifying the hosts with configuration mapreduce.job.hdfs-servers.token-renewal.exclude=<host1>,<host2>,..,<hostN>.

But seems Spark lacks of similar option. So the job submission fails if YARN fails to renew DelegationToken for any of the remote HDFS cluster. The failure in DT renewal can happen due to many reason like Remote HDFS does not trust Kerberos identity of YARN etc.

Issue Links

• relates to: YARN-3021 YARN's delegation-token handling disallows certain trust setups to operate properly over DistCp
• links to: [Github] Pull Request #31761 (viirya)
• links to: [Github] Pull Request #31761 (viirya)