[SPARK-47157] Introduce abstraction that facilitates more flexible management of file listings within the system.

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.5.0
• Fix Version/s: 4.0.0
• Component/s: Spark Core
• Labels: pull-request-available sql
• Assignee: Costas Zarifis
• Reporter: Costas Zarifis
• Created: 26/Feb/24 00:11
• Updated: 26/Feb/24 00:16
• Resolved: 26/Feb/24 00:16

Description

The introduction of these constructs is crucial for defining a standardized API for file listing operations, regardless of the underlying representation that's used to represent files and partitions. By introducing said abstractions we improve the modularity of the code and enable future improvements that can prove to be beneficial both for runtime and memory improvements.

Issue Links

• links to: GitHub Pull Request #45224