[SPARK-36394] Increase pandas API coverage in PySpark

• Type: Umbrella
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.3.0
• Component/s: PySpark
• Assignee: Xinrong Meng
• Reporter: Xinrong Meng
• Shepherd: Takuya Ueshin
• Created: 03/Aug/21 20:45
• Updated: 07/Sep/22 06:11

Description

Increase pandas API coverage in PySpark.

 

Especially, pending PRs https://github.com/databricks/koalas/pulls should be ported. Existing tickets are created for porting purposes, please avoid working on that.

Issue Links

• contains: SPARK-36395 Implement map for indexes
• relates to: SPARK-40327 Increase pandas API coverage for pandas API on Spark
• relates to: SPARK-34849 SPIP: Support pandas API layer on PySpark

Sub-Tasks

• Implement DataFrame.mode: In Progress
• Implement Series.combine: In Progress
• Implement Index.putmask: In Progress
• Implement 'weights' and 'axis' in sample at DataFrame and Series: Open
• Enable binary operations with list-like Python objects: In Progress
• Implement DataFrame.join on key column: Open
• Investigate native support for raw data containing commas: Open
• Implement __getitem__ of label-based MultiIndex: In Progress
• Add `thousands` argument to `ps.read_csv`.: In Progress
• Implement __setitem__ of label-based MultiIndex: Open
• Support Series.__and__ for Integral: In Progress
• Missing functionality in spark.pandas: Open