[SPARK-33247] Improve examples and scenarios in docstrings

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.0
• Component/s: Documentation, PySpark
• Epic Link: Project Zen
• Assignee: Unassigned
• Reporter: Hyukjin Kwon
• Created: 27/Oct/20 00:47
• Updated: 12/Dec/22 17:51
• Resolved: 24/Oct/22 11:13

Description

Currently, PySpark documentation does not have a lot of examples and scenarios. See also https://github.com/apache/spark/pull/30149#issuecomment-716490037.

We should add/improve examples especially in the commonly used APIs. For example, Column, DataFrame. RDD, SparkContext, etc.

This umbrella JIRA targets to improve them in commonly used APIs.

NOTE that we'll have to convert the docstrings into numpydoc style first in a separate PR (at SPARK-32085), and then add examples. In this way, we can manage migration to numpydoc and example improvement here separately (e.g., reverting numpydoc migration only).

Issue Links

• Dependent: SPARK-32085 Migrate to NumPy documentation style
• duplicates: SPARK-40005 Self-contained examples with parameter descriptions in PySpark documentation