[SPARK-27391] deadlock in ContinuousExecution unit tests

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 3.0.0
• Component/s: Structured Streaming
• Assignee: Jose Torres
• Reporter: Jose Torres
• Created: 04/Apr/19 21:10
• Updated: 06/Apr/19 04:16
• Resolved: 05/Apr/19 19:57

Description

ContinuousExecution (in the final query execution phrase) holds the lazy val lock of its IncrementalExecution for the entire duration of the (indefinite length) job. This can cause deadlocks in unit tests, which hook into internal APIs and try to instantiate other lazy vals.

 

(Note that this should not be able to affect production.)

Issue Links

• links to: GitHub Pull Request #24301