Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-16930

ApplicationMaster's code that waits for SparkContext is race-prone

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.1.0
    • Component/s: YARN
    • Labels:
      None

      Description

      While taking a look at SPARK-15937 and checking if there's something wrong with the code, I noticed two races that explain the behavior.

      Because they're really narrow races, I'm a little wary of declaring them the cause of that bug. Also because the logs posted there don't really explain what went wrong (and don't really look like a SparkContext was run at all).

      The races I found are:

      • it's possible, but very unlikely, for an application to instantiate a SparkContext and stop it before the AM enters the loop where it checks for the instance.
      • it's possible, but very unlikely, for an application to stop the SparkContext after the AM is already waiting for one, has been notified of its creation, but hasn't yet stored the SparkContext reference in a local variable.

      I'll fix those and clean up the code a bit in the process.

        Attachments

          Activity

            People

            • Assignee:
              vanzin Marcelo Masiero Vanzin
              Reporter:
              vanzin Marcelo Masiero Vanzin
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: