[SPARK-42883] Implement Pandas API Missing Parameters

• Type: Umbrella
• Status: Resolved
• Priority: Major
• Resolution: Resolved
• Affects Version/s: 3.4.0
• Component/s: Pandas API on Spark
• Epic Link: Pandas API Coverage Improvements
• Assignee: Xinrong Meng
• Reporter: Xinrong Meng
• Created: 21/Mar/23 09:38
• Updated: 21/Mar/23 09:40
• Resolved: 21/Mar/23 09:40

Description

pandas API on Spark aims to make pandas code work on Spark clusters without any changes. So full API coverage has been one of our major goals. Currently, most pandas functions are implemented, whereas some of them are have incomplete parameters support.

There are some common parameters missing (resolved):
 * How to do with NAs   
 * Filter data types    
 * Control result length    
 * Reindex result   

There are remaining missing parameters to implement (see doc below).

See the design and the current status at https://docs.google.com/document/d/1H6RXL6oc-v8qLJbwKl6OEqBjRuMZaXcTYmrZb9yNm5I/edit?usp=sharing.

Sub-Tasks

• EWM support ignore_na: Resolved
• Allow `columns` parameter when creating DataFrame with Series.: Resolved
• Support `ignore_index` of `Series.sort_values`: Resolved
• Enable Series.rename to change index labels: Resolved
• Implement `skipna` parameter of `DataFrame.all`: Resolved
• Implement `ignore_index` of `DataFrame.sort_index`.: Resolved
• Implement `keep` parameter of `(Index/MultiIndex).drop_duplicates`: Resolved
• Implement `dropna` parameter of `SeriesGroupBy.value_counts`: Resolved
• interpolate supports limit_area: Resolved
• Support `return_indexer` parameter of `Index/MultiIndex.sort_values`: Resolved
• Pandas API on spark Can`t apply lamda to columns.: Resolved
• Implement `ignore_index` of `DataFrame/Series.sample`: Resolved
• Implement `ignore_index` of `DataFrame.explode` and `DataFrame.drop_duplicates`: Resolved
• Implement `bool_only` parameter of `DataFrame.all` and`DataFrame.any`: Resolved
• Implement `numeric_only` parameter for `DataFrame/Series.rank` to rank numeric columns only: Resolved
• Support `na_action` and Series input correspondence in `Series.map`: Resolved
• Support string `inclusive` parameter of `Series.between`: Resolved
• Implement axis and skipna of Series.argmin: Resolved
• Implement `inplace` and `columns` parameters of `Series.drop`: Resolved
• Support `how` parameter of `MultiIndex.dropna`: Resolved
• Implement `inplace` parameter of `Series.clip`: Resolved
• Implement `skipna` of `Series.all/Index.all` to exclude NA/null values: Resolved
• Add `Series.duplicated` to indicate duplicate Series values.: Resolved
• interpolate support param `limit_direction`: Resolved
• Implement `ignore_index` of `Series.sort_values` and `Series.sort_index`: Resolved
• Implement `skipna` of `Series.argmax`: Resolved
• Support string and bool `regex` in `Series.replace`: Resolved
• Implement `keep` parameter of `frame.nlargest/nsmallest` to decide how to resolve ties: Resolved
• Adjust `GroupBy.std` to match pandas 1.4: Resolved
• Convert bools to ints in basic statistical functions of GroupBy objects: Resolved
• Implement `numeric_only` of `GroupBy.first` and `GroupBy.last`: Resolved
• Implement `skipna` parameter of `GroupBy.all`: Resolved
• Refactor `GroupBy._reduce_for_stat_function` on accepted data types: Resolved
• Implement `skipna` of basic statistical functions of DataFrame and Series: Resolved
• Adjust `GroupBy.mean/median` to match pandas 1.4: Resolved
• Implement `numeric_only` parameter of `GroupBy.max/min`: Resolved