[SPARK-32017] Make Pyspark Hadoop 3.2+ Variant available in PyPI

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.1.0
• Component/s: PySpark
• Target Version/s: 3.1.0
• Epic Link: Project Zen
• Assignee: Hyukjin Kwon
• Reporter: George Pongracz
• Created: 17/Jun/20 21:35
• Updated: 12/Dec/22 18:11
• Resolved: 23/Sep/20 00:31

Description

The version of Pyspark 3.0.0 currently available in PyPI currently uses hadoop 2.7.4.

Could a variant (or the default) have its version of Hadoop aligned to 3.2.0 as per the downloadable spark binaries.

This would enable the PyPI version to be compatible with session token authorisations and assist in accessing data residing in object stores with stronger encryption methods.

If not PyPI then as a tar file in the apache download archives at the least please.

Issue Links

• links to: [Github] Pull Request #29703 (HyukjinKwon)
• links to: [Github] Pull Request #29703 (HyukjinKwon)
• links to: [Github] Pull Request #31028 (HyukjinKwon)