[SPARK-12001] StreamingContext cannot be completely stopped if the stop() call is interrupted

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Component/s: DStreams
• Labels: backport-needed
• Assignee: Josh Rosen
• Reporter: Josh Rosen
• Created: 25/Nov/15 23:46
• Updated: 03/Dec/15 03:28
• Resolved: 02/Dec/15 21:44

Description

If StreamingContext.stop() is interrupted midway through the call, the context will be marked as stopped but certain state will have not been cleaned up. Because the stopped flag will be set, subsequent stop() calls will be unable to finish stopping the context, preventing any new StreamingContexts from being created.

We should mitigate this race condition by ensuring that the second stop() call is able to clean up the partially-stopped context.

Issue Links

• links to: [Github] Pull Request #9982 (JoshRosen)