Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-24149

Automatic namespaces discovery in HDFS federation

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.4.0
    • Fix Version/s: 2.4.0
    • Component/s: YARN
    • Labels:
      None

      Description

      Hadoop 3 introduced HDFS federation.
      Spark fails to write on different namespaces when Hadoop federation is turned on and the cluster is secure. This happens because Spark looks for the delegation token only for the defaultFS configured and not for all the available namespaces. A workaround is the usage of the property spark.yarn.access.hadoopFileSystems.

        Attachments

          Activity

            People

            • Assignee:
              mgaido Marco Gaido
              Reporter:
              mgaido Marco Gaido
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: