[SPARK-16930] ApplicationMaster's code that waits for SparkContext is race-prone

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 2.1.0
• Component/s: Spark Core, YARN
• Assignee: Marcelo Masiero Vanzin
• Reporter: Marcelo Masiero Vanzin
• Created: 05/Aug/16 23:46
• Updated: 17/May/20 18:13
• Resolved: 17/Aug/16 18:12

Description

While taking a look at SPARK-15937 and checking if there's something wrong with the code, I noticed two races that explain the behavior.

Because they're really narrow races, I'm a little wary of declaring them the cause of that bug. Also because the logs posted there don't really explain what went wrong (and don't really look like a SparkContext was run at all).

The races I found are:

• it's possible, but very unlikely, for an application to instantiate a SparkContext and stop it before the AM enters the loop where it checks for the instance.

• it's possible, but very unlikely, for an application to stop the SparkContext after the AM is already waiting for one, has been notified of its creation, but hasn't yet stored the SparkContext reference in a local variable.

I'll fix those and clean up the code a bit in the process.

Issue Links

• links to: [Github] Pull Request #14542 (vanzin)