[SPARK-23534] Spark run on Hadoop 3.0.0

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Done
• Affects Version/s: 2.3.0
• Fix Version/s: 3.0.0
• Component/s: Build
• Assignee: Unassigned
• Reporter: Saisai Shao
• Created: 28/Feb/18 08:53
• Updated: 12/Dec/22 18:10
• Resolved: 06/Dec/19 07:24

Description

Major Hadoop vendors already/will step in Hadoop 3.0. So we should also make sure Spark can run with Hadoop 3.0. This Jira tracks the work to make Spark run on Hadoop 3.0.

The work includes:

1. Add a Hadoop 3.0.0 new profile to make Spark build-able with Hadoop 3.0.
2. Test to see if there's dependency issues with Hadoop 3.0.
3. Investigating the feasibility to use shaded client jars (HADOOP-11804).

Issue Links

• duplicates: SPARK-23151 Provide a distribution of Spark with Hadoop 3.0
• is blocked by: SPARK-18673 Dataframes doesn't work on Hadoop 3.x; Hive rejects Hadoop version
• is duplicated by: SPARK-26421 Spark2.4.0 integration hadoop3.1.1 causes hive sql not to use，just in idea local mode
• is duplicated by: SPARK-26937 Build Spark 2.4 Support Hadoop-3.1 faild

Sub-Tasks

• Add Hadoop 3 profile with relevant POM fix ups: Resolved
• Provide a distribution of Spark with Hadoop 3.0: Resolved
• Make Jenkins tests passed with hadoop 3 profile: Closed
• Upgrade jackson version to 2.7.8: Resolved