[BIGTOP-2959] Add SPARK_DIST_CLASSPATH to include libraries for HDFS and YARN

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.2.1
• Fix Version/s: 1.3.0
• Component/s: build, spark
• Assignee: YoungWoo Kim
• Reporter: YoungWoo Kim
• Created: 14/Dec/17 02:44
• Updated: 27/Dec/17 02:39
• Resolved: 27/Dec/17 02:39

Description

It might be a better way to use SPARK_DIST_CLASSPATH for including the Hadoop libs instead of CLASSPATH.

export SPARK_DIST_CLASSPATH=$(hadoop classpath)

Refs. Using Spark's "Hadoop Free" Build, https://spark.apache.org/docs/2.1.1/hadoop-provided.html

Current spark-env.sh:

# Let's make sure that all needed hadoop libs are added properly
export CLASSPATH="$CLASSPATH:$HADOOP_HOME/*:$HADOOP_HDFS_HOME/*:$HADOOP_YARN_HOME/*:$HADOOP_MAPRED_HOME/*"

Issue Links

• links to: GitHub Pull Request #327