[SPARK-1318] Alignment of the Spark Shell with Spark Submit.

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.0.0
• Component/s: Spark Core
• Labels: spark-shell spark-submit
• Assignee: Unassigned
• Reporter: Bernardo Gomez Palacio
• Created: 25/Mar/14 10:43
• Updated: 09/Jul/14 20:48
• Resolved: 09/Jul/14 20:48

Description

Align the Spark-Shell with Spark-App/Spark-Submit so that the Spark-Shell offers either the same or a subset of configuration options as the Spark-App does. Ideally this should be implemented in such a way that the Spark-Shell wraps Spark-App therefore minimizing duplication.

As of now SPARK-1126 is not yet finished and there is an open question regarding how much Bash vs Scala should be leveraged. For a CLI user Bash will provide faster feedback, ergo you don't need to wait for a JVM to start. That said, from a development perspective, Scala might be a better option since we can minimize the code for scripts that target specific operating systems, mainly *NIX vs Windows.

Additional References:

• Discussion over Spark-1186 Github Pull 116.
• SPARK-1126:Spark-App
• SPARK-1186:Enrich Spark-Shell

Issue Links

• relates to: SPARK-1126 spark-submit script for running compiled binaries
• relates to: SPARK-1186 Enrich the Spark Shell to support additional arguments.