[SPARK-48649] Add "ignoreInvalidPartitionPaths" and "spark.sql.files.ignoreInvalidPartitionPaths" configs to allow ignoring invalid partition paths

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: SQL
• Labels: pull-request-available
• Assignee: Ivan Sadikov
• Reporter: Ivan Sadikov
• Created: 18/Jun/24 04:48
• Updated: 21/Jun/24 14:24
• Resolved: 19/Jun/24 01:47

Description

When having a table directory with invalid partitions such as:

table/
  invalid/...
  part=1/...
  part=2/...
  part=3/...

a SQL query reading all of the partitions would fail with 

java.lang.AssertionError: assertion failed: Conflicting directory structures detected. Suspicious paths: 
 table 
 table/invalid 

 

I propose to add a data source option and Spark SQL config to ignore invalid partition paths. The config will be disabled by default to retain the current behaviour.

spark.conf.set("spark.sql.files.ignoreInvalidPartitionPaths", "true")

spark.read.format("parquet").option("ignoreInvalidPartitionPaths", "true").load(...)  

Issue Links

• links to: GitHub Pull Request #47006