[SPARK-19464] Remove support for Hadoop 2.5 and earlier

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.1.0
• Fix Version/s: 2.2.0
• Component/s: Spark Core, YARN
• Labels: releasenotes
• Target Version/s: 2.2.0
• Assignee: Sean R. Owen
• Reporter: Sean R. Owen
• Created: 05/Feb/17 11:51
• Updated: 19/Feb/17 10:28
• Resolved: 08/Feb/17 12:20

Description

As discussed last year, and again on the mailing list this week, it's likely time to remove support for Hadoop 2.5 and earlier. Distros have been on 2.6 for over 2 years, and upstream is on to 2.7 / 3.x. It only affects Hadoop cluster users. 2.1.x continues to work on 2.2+.

The benefit is mostly cleaning up the build matrix, reasoning about dependencies across Hadoop versions, and bits of code across the project that depend on reflection to deal with newer features. It makes it a little easier to use newer Hadoop APIs. De facto, I doubt that versions before 2.6 would be given serious support in the project already anyway.

Issue Links

• is related to: SPARK-19660 Replace the configuration property names that are deprecated in the version of Hadoop 2.6
• relates to: SPARK-19545 Compilation error with method not found when build against Hadoop 2.6.0.
• links to: [Github] Pull Request #16810 (srowen)
• links to: [Github] Pull Request #16853 (srowen)
• links to: [Github] Pull Request #16858 (dongjoon-hyun)