[SPARK-26542] Support the coordinator to demerminte post-shuffle partitions more reasonably

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.2.0, 2.2.1, 2.2.2, 2.3.0, 2.3.1, 2.3.2
• Fix Version/s: 2.3.0
• Component/s: SQL
• Target Version/s: 2.3.0
• Flags: Patch
• Assignee: Unassigned
• Reporter: chenliang
• Created: 05/Jan/19 04:41
• Updated: 06/Jan/19 00:52
• Resolved: 05/Jan/19 14:35

Description

For SparkSQL ,when we open AE by 'set spark.sql.adapative.enable=true'，the ExchangeCoordinator will introduced to determine the number of post-shuffle partitions. But in some certain conditions,the coordinator performed not very well, there are always some tasks retained and they worked with Shuffle Read Size / Records 0.0B/0 ,We could increase the spark.sql.adaptive.shuffle.targetPostShuffleInputSize to solve this,but this action is unreasonable as targetPostShuffleInputSize Should not be set too large.