[SPARK-1652] Fixes and improvements for spark-submit/configs

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 1.2.0
• Component/s: Spark Core, YARN
• Assignee: Patrick Wendell
• Reporter: Patrick Wendell
• Created: 28/Apr/14 02:31
• Updated: 27/Dec/14 07:40
• Resolved: 27/Dec/14 07:40

Description

These are almost all a result of my config patch. Unfortunately the changes were difficult to unit-test and there several edge cases reported.

Issue Links

• is duplicated by: SPARK-1905 Issues with `spark-submit`

Sub-Tasks

• spark-submit for yarn prints warnings even though calling as expected: Resolved
• Spark submit should not use SPARK_CLASSPATH internally (creates warnings): Resolved
• Spark shell doesn't correctly pass quoted arguments to spark-submit: Resolved
• Executor fails to start when Command.extraJavaOptions contains multiple Java options: Resolved
• Spark submit should fail gracefully if YARN support not enabled: Resolved
• improvements spark-submit usage: Resolved
• SPARK_JAVA_OPTS and SPARK_YARN_USER_ENV are not getting propagated: Resolved
• spark-submit --name doesn't work in yarn-client mode: Resolved
• Spark on Yarn, authentication broken by pr299: Resolved
• Clean up use of setExecutorEnvs in SparkConf: Resolved
• spark.executor.extraLibraryPath isn't applied on yarn: Resolved
• spark-submit throws an exception: Exception in thread "main" java.lang.ClassNotFoundException: org.apache.spark.broadcast.TorrentBroadcastFactory: Resolved
• Support setting SPARK_JAVA_OPTS on executors for backwards compatibility: Resolved
• Add missing description to spark-env.sh.template: Resolved
• spark-submit does not set driver memory correctly: Resolved
• Jars specified via --jars in spark-submit are not added to executor classpath for YARN: Resolved
• Fix running PySpark files on YARN: Resolved
• spark-submit doesn't send master URL to Driver in standalone cluster mode: Closed
• Support local app jar in standalone cluster mode: Closed
• "--jars" is not supported in standalone cluster mode: Closed
• Allow duplicate jar files among the app jar and secondary jars in yarn-cluster mode: Resolved
• Make local:/ scheme work in more deploy modes: Resolved