[SPARK-26543] Support the coordinator to determine post-shuffle partitions more reasonably

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 2.2.0, 2.2.1, 2.2.2, 2.3.0, 2.3.1, 2.3.2
• Component/s: SQL
• Labels: bulk-closed
• Flags: Patch
• Assignee: Unassigned
• Reporter: chenliang
• Created: 05/Jan/19 04:47
• Updated: 12/Dec/22 18:10
• Resolved: 25/May/21 01:40

Description

For SparkSQL ,when we open AE by 'set spark.sql.adapative.enable=true'，the ExchangeCoordinator will introduced to determine the number of post-shuffle partitions. But in some certain conditions,the coordinator performed not very well, there are always some tasks retained and they worked with Shuffle Read Size / Records 0.0B/0 ,We could increase the spark.sql.adaptive.shuffle.targetPostShuffleInputSize to solve this,but this action is unreasonable as targetPostShuffleInputSize Should not be set too large. As follow:

We can filter the useless partition(0B) with ExchangeCoorditinator automatically

Issue Links

• links to: GitHub Pull Request #23490