[SPARK-24149] Automatic namespaces discovery in HDFS federation

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 2.4.0
• Component/s: Spark Core, YARN
• Assignee: Marco Gaido
• Reporter: Marco Gaido
• Created: 02/May/18 14:00
• Updated: 17/May/20 18:14
• Resolved: 18/May/18 20:04

Description

Hadoop 3 introduced HDFS federation.
Spark fails to write on different namespaces when Hadoop federation is turned on and the cluster is secure. This happens because Spark looks for the delegation token only for the defaultFS configured and not for all the available namespaces. A workaround is the usage of the property spark.yarn.access.hadoopFileSystems.

Issue Links

• links to: [Github] Pull Request #21216 (mgaido91)
• links to: [Github] Pull Request #21734 (wangyum)