Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-3685

Spark's local dir should accept only local paths

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.1.0
    • 2.3.0
    • Spark Core, YARN
    • None

    Description

      When you try to set local dirs to "hdfs:/tmp/foo" it doesn't work. What it will try to do is create a folder called "hdfs:" and put "tmp" inside it. This is because in Util#getOrCreateLocalRootDirs we use java.io.File instead of Hadoop's file system to parse this path. We also need to resolve the path appropriately.

      This may not have an urgent use case, but it fails silently and does what is least expected.

      Attachments

        Issue Links

          relates to

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-1529 Support DFS based shuffle in addition to Netty shuffle

          • Major - Major loss of function.
          • Resolved
          links to

          Pull request #19934 [Github] Pull Request #19934 (HyukjinKwon)

          Activity

            People

              gurwls223 Hyukjin Kwon
              andrewor14 Andrew Or
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: