[SPARK-41932] Bootstrapping Spark Connect

• Type: Test
• Status: Resolved
• Priority: Critical
• Resolution: Done
• Affects Version/s: 3.4.0
• Component/s: Connect
• Epic Link: Spark Connect
• Assignee: Hyukjin Kwon
• Reporter: Hyukjin Kwon
• Created: 07/Jan/23 05:00
• Updated: 2 days ago 23:52
• Resolved: 2 days ago 23:52

Description

We should:

1. Have an easy way to start the server. Like sbin/start-thriftserver
2. Provide an easy way to run the PySpark shell without manual server start. Like spark-sql script.

Sub-Tasks

• Provide local mode that automatically starts the server: Resolved
• Pass configurations when local remote mode is on: Resolved
• Reuses JVM initialization, and separate configuration groups to set in remote local mode: Resolved
• Local mode should work with IPython: Resolved
• Add scripts to start and stop Spark Connect server: Resolved
• Provide a mode to replace Py4J for local communication: Resolved