[SPARK-42571] Provide a mode to replace Py4J for local communication

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 3.5.0
• Fix Version/s: 4.0.0
• Component/s: Connect, PySpark
• Assignee: Hyukjin Kwon
• Reporter: Hyukjin Kwon
• Created: 25/Feb/23 00:47
• Updated: 3 days ago 23:51
• Resolved: 3 days ago 23:51

Description

We can replace Py4J even when master is specified. e.g., SparkSession.builder.master("...") and communicate w/ JVM via Spark Connect instead of Py4J.

Issue Links

• duplicates: SPARK-50605 Add spark.api.mode for better compatibility with Spark Classic