[SPARK-33212] Upgrade to Hadoop 3.2.2 and move to shaded clients for Hadoop 3.x profile

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.1
• Fix Version/s: 3.2.0
• Component/s: Spark Core, Spark Submit, SQL, YARN
• Labels: releasenotes
• Assignee: Chao Sun
• Reporter: Chao Sun
• Created: 21/Oct/20 20:47
• Updated: 22/Nov/23 12:11
• Resolved: 15/Jan/21 22:07

Description

Hadoop 3.x+ offers shaded client jars: hadoop-client-api and hadoop-client-runtime, which shade 3rd party dependencies such as Guava, protobuf, jetty etc. This Jira switches Spark to use these jars instead of hadoop-common, hadoop-client etc. Benefits include:

• It unblocks Spark from upgrading to Hadoop 3.2.2/3.3.0+. The newer versions of Hadoop have migrated to Guava 27.0+ and in order to resolve Guava conflicts, Spark depends on Hadoop to not leaking dependencies.
• It makes Spark/Hadoop dependency cleaner. Currently Spark uses both client-side and server-side Hadoop APIs from modules such as hadoop-common, hadoop-yarn-server-common etc. Moving to hadoop-client-api allows use to only use public/client API from Hadoop side.
• Provides a better isolation from Hadoop dependencies. In future Spark can better evolve without worrying about dependencies pulled from Hadoop side (which used to be a lot).

There are some behavior changes introduced with this JIRA, when people use Spark compiled with Hadoop 3.x:

• Users now need to make sure class path contains `hadoop-client-api` and `hadoop-client-runtime` jars when they deploy Spark with the `hadoop-provided` option. In addition, it is high recommended that they put these two jars before other Hadoop jars in the class path. Otherwise, conflicts such as from Guava could happen if classes are loaded from the other non-shaded Hadoop jars.
• Since the new shaded Hadoop clients no longer include 3rd party dependencies. Users who used to depend on these now need to explicitly put the jars in their class path.

Ideally the above should go to release notes.

Issue Links

• causes: SPARK-33618 hadoop-aws doesn't work
• causes: SPARK-36835 Spark 3.2.0 POMs are no longer "dependency reduced"
• causes: SPARK-36873 Add provided Guava dependency for network-yarn module
• depends upon: HADOOP-11804 Shaded Hadoop client artifacts and minicluster
• is related to: SPARK-34487 K8s integration test should use the runtime Hadoop Version
• is related to: SPARK-35323 Remove unused libraries from LICENSE-binary
• is related to: ORC-1430 Use Hadoop 3.3.5 shaded clients
• is required by: SPARK-29250 Upgrade to Hadoop 3.3.1
• relates to: HBASE-28213 Evaluate using hbase-shaded-client-byo-hadoop for Spark connector
• relates to: SPARK-35959 Add a new Maven profile "no-shaded-client" for older Hadoop 3.x versions
• links to: [Github] Pull Request #29843 (sunchao)
• links to: [Github] Pull Request #30508 (dongjoon-hyun)
• links to: [Github] Pull Request #30556 (sunchao)
• links to: [Github] Pull Request #30556 (sunchao)
• links to: [Github] Pull Request #30701 (sunchao)
• links to: [Github] Pull Request #30701 (sunchao)
• links to: [Github] Pull Request #31203 (sunchao)
• links to: [Github] Pull Request #31203 (sunchao)
• links to: [Github] Pull Request #31391 (bozhang2820)
• links to: [Github] Pull Request #31391 (bozhang2820)
• links to: [Github] Pull Request #31467 (HyukjinKwon)
• links to: [Github] Pull Request #31642 (sunchao)