[TEZ-4334] Fix deadlock in ShuffleScheduler between ShuffleScheduler.close() and the ShufflePenaltyReferee thread

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.10.3
• Assignee: Sungwoo Park
• Reporter: Sungwoo Park
• Created: 03/Sep/21 03:19
• Updated: 13/Mar/23 19:51
• Resolved: 06/Mar/23 15:32

Description

Deadlock can be generated between a thread calling ShuffleScheduler.close() and the ShufflePenaltyReferee thread.

Example (produced with an earlier version):

{{"Fetcher_O

{ attempt_1611850856294_0026_1_03_000000_0_10344 Reducer_3}

#13" #2669 daemon prio=5 os_prio=0 tid=0x00002b9de869d000 nid=0xf99 in Object.wait() [0x00002b9de4983000]
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.ShuffleScheduler.close(ShuffleScheduler.java:481)
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle.cleanupShuffleScheduler(Shuffle.java:352)
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle.cleanupShuffleSchedulerIgnoreErrors(Shuffle.java:343)
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle.reportException(Shuffle.java:407)

• locked <0x00002b96bbb9d7a8> (a org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle)
  at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.ShuffleScheduler.isShuffleHealthy(ShuffleScheduler.java:1033)
  at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.ShuffleScheduler.copyFailed(ShuffleScheduler.java:781)
• locked <0x00002b96b98a7860> (a org.apache.tez.runtime.library.common.shuffle.orderedgrouped.ShuffleScheduler)
  at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.setupConnection(FetcherOrderedGrouped.java:414)

"ShufflePenaltyReferee

{Reducer_3}

" #2645 daemon prio=5 os_prio=0 tid=0x00002b9560fae800 nid=0xf7d waiting for monitor entry [0x00002b9de733b000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.ShuffleScheduler$Referee.run(ShuffleScheduler.java:1322)

• waiting to lock <0x00002b96b98a7860> (a org.apache.tez.runtime.library.common.shuffle.orderedgrouped.ShuffleScheduler)}}

We can fix the deadlock with:

1) do not hold ShuffleScheduler.this when calling exceptionReporter.reportException()
2) remove synchronized in copyFailed()

Issue Links

• Is contained by: TEZ-4481 Prepare for Tez 0.10.3 release
• is related to: TEZ-4416 Dead lock triggered by ShuffleScheduler
• links to: GitHub Pull Request #150
• links to: GitHub Pull Request #273