[SPARK-2026] Maven "hadoop*" Profiles Should Set the expected Hadoop Version.

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 1.1.0
• Component/s: Build
• Assignee: Bernardo Gomez Palacio
• Reporter: Bernardo Gomez Palacio
• Created: 04/Jun/14 23:59
• Updated: 12/Jun/14 08:22
• Resolved: 08/Jun/14 08:25

Description

The Maven Profiles that refer to hadoopX, e.g. hadoop2.4, should set the expected hadoop.version.

e.g.

<profile>
      <id>hadoop-2.4</id>
      <properties>
        <protobuf.version>2.5.0</protobuf.version>
        <jets3t.version>0.9.0</jets3t.version>
      </properties>
</profile>

as it is suggested

<profile>
      <id>hadoop-2.4</id>
      <properties>
        <hadoop.version>2.4.0</hadoop.version>
         <yarn.version>${hadoop.version}</yarn.version>
        <protobuf.version>2.5.0</protobuf.version>
        <jets3t.version>0.9.0</jets3t.version>
      </properties>
</profile>

Builds can still define the -Dhadoop.version option but this will correctly default the Hadoop Version to the one that is expected according the profile that is selected.

e.g.

$ mvn -P hadoop-2.4,yarn clean compile

Issue Links

• is duplicated by: SPARK-1954 Make it easier to get Spark on YARN code to compile in IntelliJ