[SPARK-519] Support for Hadoop 2 distributions such as cdh4

• Type: New Feature
• Status: Resolved
• Resolution: Fixed
• Fix Version/s: 0.6.1
• Assignee: Thomas Dudziak
• Reporter: Thomas Dudziak
• Created: 16/Oct/12 15:11
• Updated: 04/May/13 19:12
• Resolved: 04/May/13 19:12

Description

Hadoop 2 unfortunately introduced backwards-incompatible changes in classes used by Spark, `JobContext` and `TaskAttemptContext` in both the `mapred` and `mapreduce` packages. In the new codebase, these classes are now interfaces and thus cannot be instantiated directly anymore. Instead, you are now required to use the `JobContextImpl` and `TaskAttemptContextImpl` classes, respectively.

This pull requests extracts the code in Spark where it uses these, into separate traits that are located in Hadoop major version specific source folders that the sbt build then includes/excludes as necessary.

In addition, for Hadoop 2, Spark now also needs to depend on the `hadoop-client` artifact.

This pull request is a new version of #264 against the 0.6 dev branch.