[SPARK-35807] Deprecate the `num_files` argument

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.2.0
• Component/s: PySpark
• Assignee: Haejoon Lee
• Reporter: Haejoon Lee
• Created: 18/Jun/21 04:06
• Updated: 20/Jul/21 02:31
• Resolved: 19/Jul/21 04:17

Description

We should deprecate the num_files argument in DataFrame.to_csv and DataFrame.to_json.

Because the behavior of num_files is not actually specify the number of files, but it specifies the number of partition.

So we should encourage users to use DataFrame.spark.repartition instead in the warning message.

Issue Links

• links to: [Github] Pull Request #33379 (itholic)
• links to: [Github] Pull Request #33379 (itholic)