[SPARK-38819] Run Pandas on Spark with Pandas 1.4.x

• Type: Umbrella
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.4.0
• Component/s: Pandas API on Spark, PySpark
• Assignee: Yikun Jiang
• Reporter: Yikun Jiang
• Created: 07/Apr/22 14:55
• Updated: 12/Dec/22 18:11

Description

This is a umbrella to track issues when pandas upgrade to 1.4.x

 

I disable the fast-failed in test, 19 failed:

https://github.com/Yikun/spark/pull/88/checks?check_run_id=5873627048

 

 

Sub-Tasks

• GroupByTest failed due to axis Length mismatch: Resolved
• Raise indexError when insert loc is out of bounds: Resolved
• series name should be preserved in series.mode(): Resolved
• iloc setitem failed due to "Cannot convert * into bool": Resolved
• test_nsmallest test failed tue to pandas 1.4.0-1.4.2 bug: Resolved
• Refresh dtype when astype("category"): Resolved
• Support Groupby positional indexing: Resolved
• test_multi_index_dtypes failed due to index mismatch: Resolved
• test_categories_setter failed due to pandas bug: Resolved
• groupby.apply doc test failed when SPARK_CONF_ARROW_ENABLED disable: Resolved
• Respect ps.concat sort parameter to follow pandas behavior: Resolved
• replace "NaN" with real "None" value in indexes in doctest: Resolved
• Add migration guide for PS behavior changes: Resolved
• Generates a new dataframe instead of operating inplace in setitem: Resolved
• '_SubTest' object has no attribute 'elapsed_time': Resolved
• Respect ``Series.concat`` sort parameter to follow 1.4.3 behavior: Resolved
• Upgrade pandas to 1.4.3: Resolved
• rename `required_same_anchor`: Resolved
• Remove `# doctest: +SKIP` of SPARK-38947/SPARK-39326 when infra dump pandas to 1.4+: Closed
• ImportError when creating pyspark.pandas document "Supported APIs" if pandas version is low.: Resolved
• Upgrade pandas to 1.4.4: Resolved