[SPARK-10458] Would like to know if a given Spark Context is stopped or currently stopping

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 1.6.0
• Component/s: Spark Core
• Target Version/s: 1.6.0
• Assignee: Madhusudanan Kandasamy
• Reporter: Matt Cheah
• Created: 04/Sep/15 23:43
• Updated: 22/Sep/15 07:04
• Resolved: 22/Sep/15 07:03

Description

I ran into a case where a thread stopped a Spark Context, specifically when I hit the "kill" link from the Spark standalone UI. There was no real way for another thread to know that the context had stopped and thus should have handled that accordingly.

Checking that the SparkEnv is null is one way, but that doesn't handle the case where the context is in the midst of stopping, and stopping the context may actually not be instantaneous - in my case for some reason the DAGScheduler was taking a non-trivial amount of time to stop.

Implementation wise I'm more or less requesting the boolean value returned from SparkContext.stopped.get() to be visible in some way. As long as we return the value and not the Atomic Boolean itself (we wouldn't want anyone to be setting this, after all!) it would help client applications check the context's liveliness.

Issue Links

• is duplicated by: SPARK-10701 Expose SparkContext#stopped flag with @DeveloperApi
• links to: [Github] Pull Request #8749 (kmadhugit)