Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
3.4.0
-
None
Description
Today, Spark represents file paths in various ways. Sometimes they are Hadoop `Path`s, sometimes they are `Path.toString`s, and sometimes they are `Path.toUri.toString`s.
This discrepancy means that Spark does not always work when user provided strings have special characters. Sometimes Spark will try to create a URI with an unescaped string; sometimes Spark will double-escape a path and try to access the wrong file.
This issue proposes a new `SparkPath` class meant to provide type safety when Spark is dealing with paths.