[SPARK-3234] SPARK_HADOOP_VERSION and SPARK_HIVE depend on deprecated make-distribution.sh command line options

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.0.2
• Fix Version/s: 1.1.0
• Component/s: Build
• Target Version/s: 1.2.0
• Assignee: Cheng Lian
• Reporter: Cheng Lian
• Created: 26/Aug/14 19:15
• Updated: 29/Aug/14 22:31
• Resolved: 29/Aug/14 22:31

Description

In make-distribution.sh, SPARK_HADOOP_VERSION is determined by --hadoop, and SPARK_HIVE is determined by --with-hive, but these two command line options are deprecated by `Dhadoop.version`/`-Phadoop??` and `-Phive` respectively. The result is that:

1. we may end up with something like spark-1.1.0-SNAPSHOT-bin-.tgz unless --name is specified explicitly
2. Datanucleus jars will not be included in the distribution folder/tarball unless we set SPARK_HIVE=true explicitly

A possible fix is to check hadoop.version and project.activeProfiles with mvn help:evaluate, similar to what we do now to determine VERSION.

Issue Links

• links to: [Github] Pull Request #2208 (liancheng)