[SPARK-42883] Implement Pandas API Missing Parameters

• Type: Umbrella
• Status: Resolved
• Priority: Major
• Resolution: Resolved
• Affects Version/s: 3.4.0
• Component/s: Pandas API on Spark
• Epic Link: Pandas API Coverage Improvements
• Assignee: Xinrong Meng
• Reporter: Xinrong Meng
• Created: 21/Mar/23 09:38
• Updated: 21/Mar/23 09:40
• Resolved: 21/Mar/23 09:40

Description

pandas API on Spark aims to make pandas code work on Spark clusters without any changes. So full API coverage has been one of our major goals. Currently, most pandas functions are implemented, whereas some of them are have incomplete parameters support.

There are some common parameters missing (resolved):
 * How to do with NAs   
 * Filter data types    
 * Control result length    
 * Reindex result   

There are remaining missing parameters to implement (see doc below).

See the design and the current status at https://docs.google.com/document/d/1H6RXL6oc-v8qLJbwKl6OEqBjRuMZaXcTYmrZb9yNm5I/edit?usp=sharing.