[SPARK-22216] Improving PySpark/Pandas interoperability

• Type: Epic
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 2.2.0
• Component/s: PySpark
• Labels: bulk-closed
• Assignee: Li Jin
• Reporter: Li Jin
• Created: 06/Oct/17 15:20
• Updated: 25/May/21 01:55
• Resolved: 25/May/21 01:45

Description

This is an umbrella ticket tracking the general effort to improve performance and interoperability between PySpark and Pandas. The core idea is to Apache Arrow as serialization format to reduce the overhead between PySpark and Pandas.

Issue Links

• incorporates: SPARK-21187 Complete support for remaining Spark data types in Arrow Converters

Sub-Tasks

• groupBy().apply() with pandas udf in pyspark: Resolved
• SPIP: Vectorized UDFs in Python: Resolved
• Simple Vectorized Python UDFs using Arrow: Closed
• Use Apache Arrow to Improve Spark createDataFrame from Pandas.DataFrame: Resolved
• User-defined window functions with pandas udf (unbounded window): Resolved
• User-defined aggregation functions with pandas udf: Resolved
• Design doc for different types of pandas_udf: Resolved
• Upgrade Arrow to version 0.8.0 and upgrade Netty to 4.1.17: Resolved
• Add function type argument to pandas_udf: Resolved
• Improve the description of Vectorized UDFs for non-deterministic cases: Resolved
• Register Scalar Vectorized UDFs for SQL Statement: Resolved
• Using pandas_udf when inputs are not Pandas's Series or DataFrame: Resolved
• Support alternative function form with group aggregate pandas UDF: Resolved
• Decrease memory consumption with toPandas() collection using Arrow: Resolved
• Change MapVector to NullableMapVector in ArrowColumnVector: Resolved
• Rename Pandas UDFs: Resolved
• Refactor group aggregate pandas UDF to its own catalyst rules: Resolved
• Pandas grouped udf on dataset with timestamp column error: Resolved
• Explicitly specify supported types in Pandas UDFs: Resolved
• Adds a conf for Arrow fallback in toPandas/createDataFrame with Pandas DataFrame: Resolved
• Improve test cases for all supported types and unsupported types: Resolved
• Explicitly check supported types in toPandas: Resolved
• Update Pandas UDFs section in sql-programming-guide: Resolved
• Support partial function and callable object with pandas UDF: Resolved
• Race condition in ArrowPythonRunner causes unclean shutdown of Arrow memory allocator: Resolved
• Pandas Grouped Map UserDefinedFunction mixes column labels: Resolved
• User-defined window functions with pandas udf (bounded window): Resolved
• Support GROUPED_AGG_PANDAS_UDF in Pivot: Resolved
• Can not mix vectorized and non-vectorized UDFs: Resolved
• Fix pandas_udf with return type StringType() to handle str type properly in Python 2.: Resolved
• Allow None for Decimal type conversion (specific to PyArrow 0.9.0): Resolved
• Show some kind of test output to indicate pyarrow tests were run: Resolved
• Improve toPandas with Arrow by sending out-of-order record batches: Resolved
• Add an example for having two columns as the grouping key in group aggregate pandas UDF: Resolved
• Register Grouped aggregate UDF Vectorized UDFs for SQL Statement: Resolved
• Clarify/Improve EvalType for grouped aggregate and window aggregate: Resolved
• Internally document type conversion between Pandas data and SQL types in Pandas UDFs: Resolved
• Update document type conversion for Pandas UDFs (pyarrow 0.13.0, pandas 0.24.2, Python 3.7): Resolved