[SPARK-1894] The default ec2 set-up ignores --driver-class-path

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 1.0.0
• Component/s: EC2
• Assignee: Andrew Or
• Reporter: Andrew Or
• Created: 21/May/14 04:10
• Updated: 05/Nov/14 10:45
• Resolved: 25/May/14 21:04

Description

If the user sets up an ec2 cluster, the scripts automatically add the following lines to conf/spark-env.sh

export SPARK_SUBMIT_LIBRARY_PATH="/root/ephemeral-hdfs/lib/native/"
export SPARK_SUBMIT_CLASSPATH=$SPARK_CLASSPATH:/root/ephemeral-hdfs/conf"

Unfortunately, these variables are exported after spark-submit parses the --driver-* flags, which also set these variables. As a result, all values set via the --driver-* flags get overridden.

The simple fix is to append to, instead of overwrite, these variables.