[SPARK-35793] Repartition before writing data source tables

• Type: Umbrella
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.2.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Yuming Wang
• Created: 17/Jun/21 07:26
• Updated: 17/Jun/21 07:26

Description

This umbrella ticket aim to track repartition before writing data source tables. It contains:

1. repartition by dynamic partition column before writing dynamic partition tables.
2. repartition before writing normal tables to avoid generating too many small files.
3. Improve local shuffle reader.

Sub-Tasks

• Repartition by dynamic partition columns before insert table: In Progress
• Improve CoalesceShufflePartitions to avoid generating small files: In Progress
• A not very elegant way to control ouput small file: In Progress
• Reduce the output partition of output stage to avoid producing small files.: In Progress