Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-27937

Revert changes introduced as a part of Automatic namespace discovery [SPARK-24149]

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.4.3
    • Fix Version/s: 3.0.0
    • Component/s: Spark Core
    • Labels:
    • Docs Text:
      Hide
      In Spark 3.0, the behavior for automatic delegation token retrieval for file systems is the same as Spark 2.3. Users need to explicitly include the URIs they want to access in the spark.kerberos.access.hadoopFileSystems configuration. The automatic discovery added in Spark 2.4 (SPARK-24149) was removed.
      Show
      In Spark 3.0, the behavior for automatic delegation token retrieval for file systems is the same as Spark 2.3. Users need to explicitly include the URIs they want to access in the spark.kerberos.access.hadoopFileSystems configuration. The automatic discovery added in Spark 2.4 ( SPARK-24149 ) was removed.

      Description

      Spark fails to launch for a valid deployment of HDFS while trying to get tokens for a logical nameservice instead of an actual namenode (with HDFS federation enabled). 

      On inspecting the source code closely, it is unclear why we were doing it and based on the context from SPARK-24149, it solves a very specific use case of getting the tokens for only those namenodes which are configured for HDFS federation in the same cluster. IMHO these are better left to the user to specify explicitly.

        Attachments

          Activity

            People

            • Assignee:
              Dhruve Ashar Dhruve Ashar
              Reporter:
              Dhruve Ashar Dhruve Ashar

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment