[SPARK-39283] Spark tasks stuck forever due to deadlock between TaskMemoryManager and UnsafeExternalSorter

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 3.0.0, 3.1.2
• Fix Version/s: 3.0.4, 3.3.0, 3.1.4, 3.2.2
• Component/s: Spark Core
• Labels: Deadlock spark3.0
• Assignee: Sandeep Pal
• Reporter: Sandeep Pal
• Created: 25/May/22 03:05
• Updated: 02/Jun/22 16:31
• Resolved: 31/May/22 23:01

Description

We are seems this deadlock between TaskMemoryManager and UnsafeExternalSorter pretty often on our workload. Sometime, the retry is successful but sometimes we have to do hacky ways to break the deadlocks such as turning down the worker machines explicitly. 

Below is the thread dump from the Spark UI showing the deadlock :

 

I believe there was a related Jira on the similar deadlock between the same threads and it was resolved. 
https://issues.apache.org/jira/browse/SPARK-27338

 

 

Issue Links

• links to: [Github] Pull Request #36680 (sandeepvinayak)