[SPARK-22216] Improving PySpark/Pandas interoperability

• Type: Epic
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.2.0
• Component/s: PySpark
• Assignee: Li Jin
• Reporter: Li Jin
• Created: 06/Oct/17 15:20
• Updated: 18/Jan/18 22:15

Description

This is an umbrella ticket tracking the general effort to improve performance and interoperability between PySpark and Pandas. The core idea is to Apache Arrow as serialization format to reduce the overhead between PySpark and Pandas.

Issue Links

• incorporates: SPARK-21187 Complete support for remaining Spark data types in Arrow Converters

Sub-Tasks

• groupBy().apply() with pandas udf in pyspark: Resolved
• SPIP: Vectorized UDFs in Python: Resolved
• Simple Vectorized Python UDFs using Arrow: Closed
• Use Apache Arrow to Improve Spark createDataFrame from Pandas.DataFrame: Resolved
• User-defined window functions with pandas udf: In Progress
• User-defined aggregation functions with pandas udf: Resolved
• Design doc for different types of pandas_udf: Resolved
• Upgrade Arrow to version 0.8.0 and upgrade Netty to 4.1.17: Resolved
• Add function type argument to pandas_udf: Resolved
• Improve the description of Vectorized UDFs for non-deterministic cases: Resolved
• Register Vectorized UDFs for SQL Statement: Resolved
• Using pandas_udf when inputs are not Pandas's Series or DataFrame: Resolved
• Support alternative function form with group aggregate pandas UDF: Resolved
• Decrease memory consumption with toPandas() collection using Arrow: Open
• Change MapVector to NullableMapVector in ArrowColumnVector: Resolved
• Rename Pandas UDFs: Resolved
• Refactor group aggregate pandas UDF to its own catalyst rules: Open
• Pandas grouped udf on dataset with timestamp column error: Resolved
• Explicitly specify supported types in Pandas UDFs: Resolved
• Adds a conf for Arrow fallback in toPandas/createDataFrame with Pandas DataFrame: Resolved
• Improve test cases for all supported types and unsupported types: Open
• Explicitly check supported types in toPandas: Resolved
• Update Pandas UDFs section in sql-programming-guide: Open
• Support partial function and callable object with pandas UDF: Open