[SPARK-35805] API auditing in Pandas API on Spark

• Type: Umbrella
• Status: Resolved
• Priority: Blocker
• Resolution: Done
• Affects Version/s: 3.2.0
• Component/s: PySpark
• Target Version/s: 3.2.0
• Epic Link: Project Zen
• Assignee: Haejoon Lee
• Reporter: Haejoon Lee
• Created: 18/Jun/21 04:00
• Updated: 30/Jul/21 00:18
• Resolved: 30/Jul/21 00:18

Description

There are several things that need improvement in pandas on Spark.

Issue Links

• is part of: SPARK-34849 SPIP: Support pandas API layer on PySpark

Sub-Tasks

• Mapping the `mode` argument to pandas in DataFrame.to_csv: Resolved
• Deprecate the `num_files` argument: Resolved
• Always enable the `pandas_metadata` in DataFrame.parquet: Resolved
• Add `index_col` argument for ps.sql.: Resolved
• Deprecate ps.broadcast API: Resolved
• Deprecate DataFrame.to_spark_io: Resolved
• Throw an error if `version` and `timestamp` are used together in DataFrame.to_delta.: Resolved
• Remove some APIs from documentation.: Resolved
• Install mlflow/sklearn in Github Actions CI: Resolved