[SPARK-17976] Global options to spark-submit should not be position-sensitive

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Not A Problem
• Affects Version/s: 2.0.0, 2.0.1
• Component/s: Deploy
• Assignee: Unassigned
• Reporter: Nicholas Chammas
• Created: 17/Oct/16 21:09
• Updated: 17/Oct/16 21:25
• Resolved: 17/Oct/16 21:25

Description

It is maddening that this does what you expect:

spark-submit --packages graphframes:graphframes:0.2.0-spark2.0-s_2.11 \
    file.py 

whereas this doesn't because --packages is totally ignored:

spark-submit file.py \
    --packages graphframes:graphframes:0.2.0-spark2.0-s_2.11

Ideally, global options should be valid no matter where they are specified.

If that's too much work, then I think at the very least spark-submit should display a warning that some input is being ignored. (Ideally, it should error out, but that's probably not possible for backwards-compatibility reasons at this point.)