[TEZ-3297] Deadlock scenario in AM during ShuffleVertexManager auto reduce

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 0.7.2, 0.9.0, 0.8.4
• Target Version/s: 0.7.2, 0.9.0, 0.8.4
• Hadoop Flags: Reviewed
• Assignee: Rajesh Balamohan
• Reporter: Zhiyuan Yang
• Created: 09/Jun/16 22:15
• Updated: 22/Aug/17 00:02
• Resolved: 13/Jun/16 00:06

Description

Here is what's happening in the attached thread dump.

App Pool thread #9 does the auto reduce on V2 and initializes the new edge manager, it holds the V2 write lock and wants read lock of source vertex V1.

At the same time, another App Pool thread #2 schedules a task of V1 and gets the output spec, so it holds the V1 read lock and wants V2 read lock.

Also, dispatcher thread wants the V1 write lock to begin the state machine transition. Since dispatcher thread is at the head of V1 ReadWriteLock queue, thread #9 cannot get V1 read lock even thread #2 is holding V1 read lock.

This is a circular lock scenario. #2 blocks dispatcher, dispatcher blocks #9, and #9 blocks #2.

There is no problem with ReadWriteLock behavior in this case. Please see this java bug report, http://bugs.java.com/bugdatabase/view_bug.do?bug_id=6816565.