Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-22372

Make YARN client extend SparkApplication

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.3.0
    • Fix Version/s: 2.3.0
    • Component/s: YARN
    • Labels:
      None

      Description

      For SPARK-11035 to work well, at least in cluster mode, YARN needs to implement SparkApplication so that it doesn't use system properties to propagate Spark configuration from spark-submit.

      There is a second complication, that YARN uses system properties to propagate SPARK_YARN_MODE on top of other Spark configs. We should take a look at either change that to a configuration, or remove SPARK_YARN_MODE altogether if possible.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                vanzin Marcelo Vanzin
                Reporter:
                vanzin Marcelo Vanzin
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: