[TEZ-4416] Dead lock triggered by ShuffleScheduler

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 0.10.1
• Fix Version/s: 0.10.3
• Target Version/s: 0.10.1
• Flags: Important

Language

JAVA

• Assignee: Unassigned
• Reporter: Omega-Ariston
• Created: 25/May/22 08:33
• Updated: 07/Mar/23 09:03
• Resolved: 07/Mar/23 09:03

Description

How this bug is found:

I was executing a sql with Hive on tez on a cluster that has low disk capacity. An exception was thrown during the execution (which is quite reasonable). Yet the task didn't stop normally, but keep hanging there for a very long while. Therefore, I printed out the jstack and did some investigation. Here's what I found.

(The .jstack file and the screenshot of  jstack segment are attached below.)

 

How this dead lock is triggered:

1. Fail to copy files on local disk, which will trigger copyFailed() from FetcherOrderedGrouped.copyFromHost(), which is a synchronized method on ShuffleScheduler instance. 
2. Method called from 1 will eventually goes to ShuffleScheduler.close(), in which it tries to kill the Referee's thread by calling referee.interrupt() and referee.join().
3. Meanwhile, Referee is waiting for ShuffleScheduler's instance lock in its run() method, which is hold by the process from 1. Hence a dead lock happens.

Issue Links

• relates to: TEZ-4334 Fix deadlock in ShuffleScheduler between ShuffleScheduler.close() and the ShufflePenaltyReferee thread