[SPARK-49501] Catalog createTable API is double-escaping paths

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0, 3.5.4
• Component/s: Spark Core
• Labels: pull-request-available
• Target Version/s: 4.0.0
• Assignee: Christos Stavrakakis
• Reporter: Christos Stavrakakis
• Shepherd: Wenchen Fan
• Created: 03/Sep/24 14:17
• Updated: 09/Sep/24 09:28
• Resolved: 09/Sep/24 09:25

Description

Creating an external table using spark.catalog.createTable results in incorrect escaping of special chars in paths.

Consider the following code:

{{spark.catalog.createTable("testTable", source = "parquet", schema = new StructType().add("id", "int"), description = "", options = Map("path" -> "/tmp/test table"))}}

The above call creates a table that is stored in /tmp/test%20table instead of /tmp/test table. Note that this behaviour is different from the SQL API, e.g. create table testTable(id int) using parquet location '/tmp/test table'

Issue Links

• links to: GitHub Pull Request #47976