[SPARK-45776] Remove the defensive null check added in SPARK-39553.

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: Spark Core
• Labels: pull-request-available
• Assignee: Yang Jie
• Reporter: Yang Jie
• Created: 03/Nov/23 03:45
• Updated: 03/Nov/23 09:59
• Resolved: 03/Nov/23 09:57

Description

def unregisterShuffle(shuffleId: Int): Unit = {
    shuffleStatuses.remove(shuffleId).foreach { shuffleStatus =>
      // SPARK-39553: Add protection for Scala 2.13 due to https://github.com/scala/bug/issues/12613
      // We should revert this if Scala 2.13 solves this issue.
      if (shuffleStatus != null) {
        shuffleStatus.invalidateSerializedMapOutputStatusCache()
        shuffleStatus.invalidateSerializedMergeOutputStatusCache()
      }
    }
  } 

This issue has been fixed in Scala 2.13.9.

Issue Links

• links to: GitHub Pull Request #43644