Uploaded image for project: 'Livy'
  1. Livy
  2. LIVY-102

Don't leave spark-submit launchers lying around

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 0.2
    • 0.2
    • Core, RSC
    • None

    Description

      When Livy launches Spark in cluster mode, the spark-submit process used to launch Spark hangs around in the background, consuming resources and printing lots of things to the log.

      Instead, we should let the process go away since it's not useful anymore after the app is started. That's the spark.yarn.submit.waitAppCompletion config option (default = true), but we also need to make sure the RSC (and batch sessions) won't detect that as an error.

      Attachments

        Activity

          People

            vanzin Marcelo Masiero Vanzin
            vanzin Marcelo Masiero Vanzin
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: