[SPARK-44999] Refactor ExternalSorter to reduce checks on shouldPartition when calling getPartition

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: Spark Core
• Assignee: Yang Jie
• Reporter: Yang Jie
• Created: 29/Aug/23 03:39
• Updated: 29/Aug/23 15:30
• Resolved: 29/Aug/23 15:30

Description

  private def getPartition(key: K): Int = {
    if (shouldPartition) partitioner.get.getPartition(key) else 0
  } 

 

The getPartition method checks shouldPartition every time it is called. However, shouldPartition should not be able to change after the ExternalSorter is instantiated. Therefore, it can be refactored to reduce the checks on shouldPartition.

Issue Links

• links to: [Github] Pull Request #42713 (LuciferYang)
• links to: [Github] Pull Request #42713 (LuciferYang)