[SPARK-46910] Eliminate JDK Requirement in PySpark Installation

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.5.0
• Fix Version/s: 4.0.0
• Component/s: PySpark
• Labels: pull-request-available
• Assignee: Amanda Liu
• Reporter: Amanda Liu
• Created: 29/Jan/24 22:54
• Updated: 30/Jan/24 00:48
• Resolved: 30/Jan/24 00:48

Description

PySpark requires users to have the correct JDK version (JDK 8+ for Spark<4; JDK 17+ for Spark>=4) installed locally.

We can make the Spark installation script install the JDK, so users don’t need to do this step manually.

Details

1. When the entry point for a Spark class is invoked, the spark-class script checks if Java is installed in the user environment.

1. If Java is not installed, the user is prompted to select whether they want to install JDK 17.

1. If the user selects yes, JDK 17 is installed (using the install-jdk library) and JAVA_HOME variable and RUNNER are set appropriately. The Spark build will now work!

1. If the user selects no, we provide them a brief description of how to install JDK manually.

Issue Links

• links to: GitHub Pull Request #44940