[SPARK-49519] Merge options of table and relation when create FileScanBuilder

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: SQL
• Labels: pull-request-available
• Assignee: Jiayi Liu
• Reporter: Jiayi Liu
• Created: 05/Sep/24 02:47
• Updated: 11/Sep/24 14:43
• Resolved: 11/Sep/24 14:42

Description

Currently, the subclass of FileTable only uses the options from relation when constructing the FileScanBuilder, which leads to the omission of the contents in FileTable.options. For the TableCatalog, the dsOptions can be set into the FileTable.options returned by the TableCatalog.loadTable method. If only the relation options are used here, the TableCatalog will not be able to pass dsOptions that contains table options to FileScan.

Merge the two options is a better option.
 
 

Issue Links

• links to: GitHub Pull Request #47996