[SPARK-27627] Make option "pathGlobFilter" as a general option for all file sources - ASF JIRA

Attach files

Attach Screenshot

Voters

Watch issue

Watchers

Create sub-task

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Improvement
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: 3.0.0
Fix Version/s: 3.0.0
Component/s: SQL
Labels:
None

Description

Background:
The data source option "pathGlobFilter" is introduced for Binary file format: https://github.com/apache/spark/pull/24354 , which can be used for filtering file names, e.g. reading ".png" files only while there is ".json" files in the same directory.

Proposal:
Make the option "pathGlobFilter" as a general option for all file sources. The path filtering should happen in the path globbing on Driver.

Motivation:
Filtering the file path names in file scan tasks on executors is kind of ugly.

Impact:
1. The splitting of file partitions will be more balanced.
2. The metrics of file scan will be more accurate.
3. Users can use the option for reading other file sources.