[SPARK-41949] Make stage scheduling support local-cluster mode

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: PySpark, Spark Core
• Assignee: Weichen Xu
• Reporter: Weichen Xu
• Created: 09/Jan/23 09:08
• Updated: 10/Jan/23 13:06
• Resolved: 10/Jan/23 13:06

Description

Make stage scheduling support local-cluster mode.

This is useful in testing, especially for test code of third-party python libraries that depends on pyspark, many tests are written with pytest, but pytest is hard to integrate with a standalone spark cluster.

Issue Links

• links to: [Github] Pull Request #39424 (WeichenXu123)