[SPARK-32085] Migrate to NumPy documentation style

• Type: Umbrella
• Status: Resolved
• Priority: Major
• Resolution: Done
• Affects Version/s: 3.1.0
• Fix Version/s: 3.1.0
• Component/s: PySpark
• Epic Link: Project Zen
• Assignee: Maciej Szymkiewicz
• Reporter: Hyukjin Kwon
• Created: 24/Jun/20 08:43
• Updated: 12/Dec/22 18:10
• Resolved: 05/Jan/21 05:55

Description

https://github.com/numpy/numpydoc

For example,

Before: https://github.com/apache/spark/blob/f0e6d0ec13d9cdadf341d1b976623345bcdb1028/python/pyspark/sql/dataframe.py#L276-L318
After: https://github.com/databricks/koalas/blob/6711e9c0f50c79dd57eeedb530da6c4ea3298de2/databricks/koalas/frame.py#L1122-L1176

We can incrementally start to switch.

NOTE that this JIRA targets only to switch the style. It does not target to add additional information or fixes together.

Issue Links

• contains: SPARK-33242 Install numpydoc in Jenkins machines
• Dependent: SPARK-33247 Improve examples and scenarios in docstrings

Sub-Tasks

• Add numpydoc into documentation dependency: Resolved
• Migration to NumPy documentation style in SQL (pyspark.sql.*): Resolved
• Migration to NumPy documentation style in ML (pyspark.ml.*): Resolved
• Migration to NumPy documentation style in MLlib (pyspark.mllib.*): Resolved
• Migration to NumPy documentation style in Streaming (pyspark.streaming.*): Resolved
• Migration to NumPy documentation style in Core (pyspark.*, pyspark.resource.*, etc.): Resolved
• Update contribution guide about NumPy documentation style: Resolved