[SPARK-32935] File source V2: support bucketing

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Gengliang Wang
• Created: 18/Sep/20 08:07
• Updated: 18/Sep/20 14:18

Description

Datasource V2 does not currently support bucketed reads or writes similar to Datasource V1 does.  See DatasourceScanExec and config 

spark.sql.sources.bucketing.enabled.  We need to add support to V2 as well.

 

Support writing file data source with bucketing looks like:

 
fileDf.write.bucketBy(...).sortBy(..)...