[SPARK-44728] Improve PySpark documentations

• Type: Umbrella
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.5.0, 4.0.0
• Fix Version/s: 4.0.0
• Component/s: PySpark
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: Allison Wang
• Created: 08/Aug/23 22:59
• Updated: 07/Nov/24 20:54
• Resolved: 10/Jul/24 00:56

Description

An umbrella Jira ticket to improve the PySpark documentation.
 
 

Issue Links

• is related to: SPARK-44111 Prepare Apache Spark 4.0.0
• links to: GitHub Pull Request #47269
• links to: GitHub Pull Request #47345

Sub-Tasks

• Add canonical links to the PySpark docs page: Resolved
• Add Spark version drop down to the PySpark doc site: Resolved
• Switch languages consistently across docs for all code snippets: Resolved
• Some directories should be cleared when regenerating files: Resolved
• There should be a gap at the bottom of the HTML: Resolved
• Align example order (Python -> Scala/Java -> R) in all Spark Doc Content: Resolved
• Refine docstring of `DataFrame.drop`: Resolved
• Refine docstring of `groupBy/rollup/cube`: Resolved
• Refine docstring of `ceil/ceiling/floor/round/bround`: Resolved
• Refine docstring of `rand/randn`: Resolved
• Add Matomo analytics to all released docs pages: Resolved
• Supplementary exception class: Resolved
• Make code block copyable: Resolved
• Add user guide for type mappings between Spark and Python data types: Resolved
• Add documentation for type casting rules in Python UDFs/UDTFs: Open
• Make Python the first language in all Spark code snippet: Resolved
• Refine DocString of `Union*`: Resolved
• Refine docstring of `DataFrame.columns` property: Resolved
• Refine docstring of `DataFrame.isEmpty`: Resolved
• Refine docstring of `createDataFrame`: Resolved
• Refine the docstring of `DataFrame.collect`: Resolved
• Fix wildcard import `from pyspark.sql.functions import *` in `Quick Start` Examples: Resolved
• Fix docstring of `monotonically_increasing_id`: Resolved
• Enable Doctests of `rand`, `randn` and `log`: Resolved
• Refine docstring of `approx_count_distinct`: Resolved
• Refine docstring for DataFrame.approxQuantile: Resolved
• Refine docstring of `DataFrame.filter`: Resolved
• Refine docstring of `asc/desc`: Resolved
• Refine docstring of `Column.between`: Resolved
• Refine DocStrings of `try_{add, subtract, multiply, divide, avg, sum}`: Resolved
• Refine docstring of `max`: Resolved
• Refine docstring of `DataFrame.distinct`: Resolved
• Refine docstrings of `coalesce/repartition/repartitionByRange`: Resolved
• Refine docstrings of `min_by/max_by`: Resolved
• Refine docstring of `min`: Resolved
• Refine docstring of `explode`: Resolved
• Refine docstrings of `collect_list/collect_set`: Resolved
• Adjust the `versionadded` and `versionchanged` information to the parameters: Resolved
• Refine docstring of `inline`: Resolved
• Automate updating versions. json: Open
• XML: Refine docstring of schema_of_xml: Resolved
• Update Example with docker official image: Resolved
• Refine docstring of `array/array_contains/arrays_overlap`: Resolved
• Refine docstring of `Column.isin`: Resolved
• Refine docstring of `DataFrame.withColumnRenamed`: Resolved
• Refine docstring of `DataFrame.join`: Resolved
• Refine docstring of `DataFrameReader.parquet`: Resolved
• Refine docstring of `DataFrameReader.json`: Resolved
• Refine docstring of `Column.when`: Resolved
• Refine DocString of `regr_*` functions: Resolved
• Refine docstring of `sum`: Resolved
• Refine docstring of `count`: Resolved
• Refine docstring of count_distinct: Resolved
• Configurable error when generating Python docs: Open
• python function categories should be consistent with SQL function groups: Resolved
• Refine docstring of `create_map/slice/array_join`: Resolved
• Refine docstring of `DataFrame.show`: Resolved
• Refine docstring of `options` for dataframe reader and writer: Resolved
• Refine docstring of `SparkSession.builder.config`: Resolved
• Refine docstring of `lit`: Resolved
• XML: Refine docstring of from_xml: Resolved
• Add user guide for dataframe creation: Open
• Add a self-contained example about creating dataframe from jdbc: Open
• Add user guide for basic dataframe operations: Open
• Add user guide for column selections: Open
• Add user guide for groupby and aggregate: Open
• Add user guide for window operations: Open
• Refine docstring of `mapInPandas`: Resolved
• Use built-in math constant in math functions: Resolved
• Refine docstring of `DataFrame.substract`: Resolved
• Refine docstring of `DataFrame.intersectAll`: Resolved
• Refine docstring of `DataFrame.intersect`: Resolved
• Refine docstring of `DataFrame.dropna/fillna/replace`: Resolved
• Improve basic datasource examples: Resolved
• Document parameters and examples for RuntimeConf get, set and unset: Resolved
• Refine docstring of UDTF: Resolved
• Refine docstring of `concat/array_position/element_at/try_element_at`: Resolved
• Add missing `toDegrees/toRadians/atan2/approxCountDistinct` function descriptions: Resolved
• Correct the typing of schema_of_{csv, json, xml}: Resolved
• Refine docstring of `array_prepend/array_append/array_insert`: Resolved
• Refine docstring of `array_intersect/array_union/array_except`: Resolved
• Refine docstring of `array_compact/array_distinct/array_remove`: Resolved
• Refine docstring of `array_min/array_max/array_size/array_repeat`: Resolved
• Refine docstring of `get/array_zip/sort_array`: Resolved
• Refine docstring of `flatten/sequence/shuffle`: Resolved
• Refine docstring for DataFrame.createTempView/createOrReplaceTempView: Resolved
• Refine docstring for DataFrame.createGlobalTempView/createOrReplaceGlobalTempView: Resolved
• Refine docstring for DataFrame.schema/explain/printSchema: Resolved
• Refine docstring `reverse/map_contains_key`: Resolved
• Refine docstring of `map_from_arrays/map_from_entries/map_concat`: Resolved
• Refine docstring of `parse_url/url_encode/url_decode`: Resolved
• Refine docstring of `convert_timezone/make_dt_interval/make_interval`: Resolved
• Refine docstring `make_timestamp/make_timestamp_ltz/make_timestamp_ntz/make_ym_interval`: Resolved
• Refine docstring of `from_csv/schema_of_csv/to_csv`: Resolved
• Refine docstring `aes_encrypt/aes_decrypt/try_aes_decrypt`: Resolved
• Refine docstring of `map_keys/map_values/map_entries`: Resolved
• Refine docstring of `str_to_map/map_filter/map_zip_with`: Resolved
• Refine docstring of `abs/acos/acosh`: Resolved
• Refine docstring of `bit_and/bit_or/bit_xor`: Resolved
• Refine docstring of `sum_distinct/array_agg/count_if`: Resolved
• Refine docstring of `asc_nulls_first/asc_nulls_last/desc_nulls_first/desc_nulls_last`: Resolved
• Refine docstring of `to_json/from_json`: Resolved
• Refine docstring of `try_sum`, `try_avg`, `avg`, `sum`, `mean`: Resolved
• Refine docstrings of try_*: Resolved
• Improve docstring of mapInPandas: Resolved
• Update the table-valued function documentation: Resolved
• Add examples section header to `format_number` docstring: Resolved
• Improve clarity in lag docstring: Resolved
• Unify the 'See Also' section formatting across PySpark docstrings: Resolved
• Update documentation to add `column` as alias of `col`: Open
• Fix the incorrect namings and missing params in func docs in `builtin.py`: Resolved
• Test the default column name of array functions: Resolved
• Add doctest for `options` in json functions: Resolved
• Refine the type hints in functions: Resolved
• Document the NaN handling in df.na.drop: Resolved
• Fix type hint for `accuracy` in `percentile_approx` and `approx_percentile`: Resolved
• Remove experimental API notes for pandas related functions: Resolved
• Refine docstring for trigonometric functions: Resolved
• Refine docstring for basic functions: Resolved
• Refine the docstring for more date functions: Resolved
• Refine the docstring of multiple datetime functions: Resolved
• Refine docstrings for aggregation functions - part 1: Resolved