Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-6506

python support yarn cluster mode requires SPARK_HOME to be set

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.3.0
    • 1.3.2, 1.4.0
    • YARN
    • None

    Description

      We added support for python running in yarn cluster mode in https://issues.apache.org/jira/browse/SPARK-5173, but it requires that SPARK_HOME be set in the environment variables for application master and executor. It doesn't have to be set to anything real but it fails if its not set. See the command at the end of: https://github.com/apache/spark/pull/3976

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. SPARK-6701 Flaky test: o.a.s.deploy.yarn.YarnClusterSuite Python application

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-5173 support python application running on yarn cluster mode

          • Major - Major loss of function.
          • Closed
          links to

          Pull request #5405 [Github] Pull Request #5405 (vanzin)

          Activity

            People

              vanzin Marcelo Masiero Vanzin
              tgraves Thomas Graves
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: