[SPARK-38800] Explicitly document the supported pandas version.

• Type: Test
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.3.0
• Component/s: Documentation, PySpark
• Assignee: Haejoon Lee
• Reporter: Haejoon Lee
• Created: 06/Apr/22 05:12
• Updated: 12/Dec/22 18:10
• Resolved: 08/Apr/22 02:12

Description

pandas has different behavior per each version for some APIs.

So, we should explicitly follow one pandas version for one pandas-on-Spark version and document it.

For example, if some APIs follow the behavior of pandas 1.3 whereas some APIs follow the behavior of pandas 1.4, users may be confused.

Issue Links

• links to: [Github] Pull Request #36095 (itholic)
• links to: [Github] Pull Request #36114 (itholic)
• links to: [Github] Pull Request #36114 (itholic)