[SPARK-7870] Publish jars against hadoop 1 (default) too

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 1.3.2, 1.4.0
• Component/s: Build
• Labels: hadoop-1.0 hadoop-2.0 jars publish
• Assignee: Unassigned
• Reporter: Andy Petrella
• Created: 26/May/15 14:25
• Updated: 26/May/15 14:29
• Resolved: 26/May/15 14:29

Description

The published jars were always (or at least since 1.1.0, see SPARK-3764) published using hadoop 2 binaries.

However, the default are hadoop 1. Which means that an application that doesn't use hdfs will fail at simple functions like `saveAsObjectFile` (since it uses hadoop interfaces).

There are a couple of points that can be considered wrong here:

• it's not well stated that the published binaries aren't what you can expect base on spark rules w.r.t hadoop 1.0.4
• if an application doesn't need hdfs, the dev couldn't rely on public libraries, but he has to build it's own set without asking for a specific hadoop version (hence will get the 1.0.4) and deploy it on his own :-/