Details
-
Umbrella
-
Status: Open
-
Major
-
Resolution: Unresolved
-
3.2.0
-
None
-
None
Description
This umbrella ticket aim to track repartition before writing data source tables. It contains:
- repartition by dynamic partition column before writing dynamic partition tables.
- repartition before writing normal tables to avoid generating too many small files.
- Improve local shuffle reader.
Attachments
1.
|
Repartition by dynamic partition columns before insert table |
|
In Progress | Unassigned |
2.
|
Support repartition expand partitions in AQE |
|
Resolved | XiDuo You |
3.
|
Coalesce small output files through AQE |
|
Resolved | Yuming Wang |
4.
|
Improve CoalesceShufflePartitions to avoid generating small files |
|
In Progress | Unassigned |
5.
|
A not very elegant way to control ouput small file |
|
In Progress | Unassigned |
6.
|
Add a new operator to distingush if AQE can optimize safely |
|
Resolved | XiDuo You |
7.
|
Collapse above RebalancePartitions |
|
Closed | Yuming Wang |
8.
|
Only use local shuffle reader when REBALANCE_PARTITIONS_BY_NONE without CustomShuffleReaderExec |
|
Resolved | XiDuo You |
9.
|
Reduce the output partition of output stage to avoid producing small files. |
|
In Progress | Unassigned |
10.
|
Support specify initial partition number for rebalance |
|
Resolved | XiDuo You |