[SPARK-23349] Duplicate and redundant type determination for ShuffleManager Object

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Won't Fix
• Affects Version/s: 2.2.1
• Component/s: Shuffle, Spark Core, SQL
• Assignee: Unassigned
• Reporter: Phoenix_Daddy
• Created: 07/Feb/18 08:40
• Updated: 17/May/20 18:30
• Resolved: 07/Feb/18 21:47

Description

org.apache.spark.sql.execution.exchange.ShuffleExchange

In the "needtocopyobjectsbeforguffle()" function，there is a nested "if or else" branch .
The <sortBasedShufffleOn> condition in the first layer "if" has the same value as the <bypassIsSupported> condition in the second layer "if", that is, <bypassIsSupported> must be true when <sortBasedShufffleOn> is true.
In addition, the <byPassMergeThreshold> condition will be used in the second layer "if" and should not be calculated until needed.

Issue Links

• links to: [Github] Pull Request #20526 (wujianping10043419)
• links to: [Github] Pull Request #20530 (wujianping10043419)