[SPARK-31851] Redesign PySpark documentation

• Type: Umbrella
• Status: Resolved
• Priority: Critical
• Resolution: Done
• Affects Version/s: 3.1.0
• Fix Version/s: 3.1.0
• Component/s: ML, PySpark, Spark Core, SQL, Structured Streaming
• Target Version/s: 3.1.0
• Epic Link: Project Zen
• Assignee: Hyukjin Kwon
• Reporter: Hyukjin Kwon
• Created: 28/May/20 06:10
• Updated: 12/Dec/22 18:11
• Resolved: 04/Jan/21 05:44

Description

Currently, PySpark documentation (https://spark.apache.org/docs/latest/api/python/index.html) is pretty much poorly written compared to other projects.
See, for example, see Koalas https://koalas.readthedocs.io/en/latest/ as an example.

PySpark is being more and more important in Spark, and we should improve this documentation so people can easily follow.

Reference:

• https://koalas.readthedocs.io/en/latest/
• https://pandas.pydata.org/docs/

Issue Links

• contains: SPARK-32185 User Guide - Monitoring
• contains: SPARK-32391 Install pydata_sphinx_theme in Jenkins machines
• is duplicated by: SPARK-31852 Redesign PySpark documentation
• is related to: SPARK-32204 Binder Integration
• is related to: SPARK-32584 Exclude _images and _sources that are generated by Sphinx in Jekyll build