[SPARK-34544] pyspark toPandas() should return pd.DataFrame

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.1
• Fix Version/s: 3.3.0
• Component/s: PySpark
• Assignee: Maciej Szymkiewicz
• Reporter: Rafal Wojdyla
• Created: 25/Feb/21 16:01
• Updated: 12/Dec/22 18:11
• Resolved: 23/Dec/21 00:16

Description

Right now toPandas() returns DataFrameLike, which is an incomplete "view" of pandas DataFrame. Which leads to cases like mypy reporting that certain pandas methods are not present in DataFrameLike, even tho those methods are valid methods on pandas DataFrame, which is the actual type of the object. This requires type ignore comments or asserts.

Issue Links

• is duplicated by: SPARK-34540 Add convert_dtypes to the DataFrameLike protocol
• links to: [Github] Pull Request #34927 (zero323)
• links to: [Github] Pull Request #35029 (HyukjinKwon)
• links to: [Github] Pull Request #35029 (HyukjinKwon)