[SPARK-40235] Use interruptible lock instead of synchronized in Executor.updateDependencies()

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: Spark Core
• Assignee: Josh Rosen
• Reporter: Josh Rosen
• Created: 26/Aug/22 21:07
• Updated: 29/Aug/22 23:50
• Resolved: 29/Aug/22 23:50

Description

This patch modifies the synchronization in Executor.updateDependencies() in order to allow tasks to be interrupted while they are blocked and waiting on other tasks to finish downloading dependencies.

This synchronization was added years ago in mesos/spark@7b9e96c in order to prevent concurrently-launching tasks from performing concurrent dependency updates. If one task is downloading dependencies, all other newly-launched tasks will block until the original dependency download is complete.

Let's say that a Spark task launches, becomes blocked on a updateDependencies() call, then is cancelled while it is blocked. Although Spark will send a Thread.interrupt() to the canceled task, the task will continue waiting because threads blocked on a synchronized won't throw an InterruptedException in response to the interrupt. As a result, the blocked thread will continue to wait until the other thread exits the synchronized block. 

In the wild, we saw a case where this happened and the thread remained blocked for over 1 minute, causing the TaskReaper to kick in and self-destruct the executor.

This PR aims to fix this problem by replacing the synchronized with a ReentrantLock, which has a lockInterruptibly method.

Issue Links

• is related to: SPARK-40263 Use interruptible lock instead of synchronized in TransportClientFactory.createClient()
• links to: [Github] Pull Request #37681 (JoshRosen)