[SPARK-40576] Support pandas 1.5.x.

• Type: Umbrella
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: Pandas API on Spark
• Assignee: Haejoon Lee
• Reporter: Haejoon Lee
• Created: 27/Sep/22 06:41
• Updated: 27/Mar/23 02:49
• Resolved: 27/Mar/23 02:49

Description

Since the pandas version 1.5.0 is released from Sep 19, 2022, we should support the same behavior with latest pandas.

Issue Links

• is a parent of: SPARK-40966 FIX `read_parquet` with `pandas_metadata`
• links to: [Github] Pull Request #37955 (itholic)
• links to: [Github] Pull Request #37955 (itholic)

Sub-Tasks

• Fix CategoricalIndex.append: Resolved
• Fix `IndexesTest.test_to_frame` when pandas 1.5.0: Resolved
• Upgrade pandas to 1.5.0: Resolved
• Update the document for DataFrame.to_orc: Resolved
• Fix test for `DataFrame.corr_with` to skip the pandas regression: Resolved
• Fix `ps.read_parquet` when pandas_metadata is True: Resolved
• Fix plotting features work properly with pandas 1.5.0.: Resolved
• Fix `corrwith` to work properly with different anchor.: Resolved
• Re-enable the DataFrame.corrwith test after fixing in future pandas.: Resolved
• Fix doctest for `Index.(isin|isnull|notnull)` to work properly with pandas 1.5: Resolved
• Upgrade pandas to 1.5.1: Resolved
• FIX `read_parquet` with `pandas_metadata`: Resolved