[SPARK-22216] Improving PySpark/Pandas interoperability

• Type: Epic
• Status: Reopened
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.2.0
• Component/s: PySpark
• Assignee: Li Jin
• Reporter: Li Jin
• Created: 06/Oct/17 15:20
• Updated: 21/Jun/19 06:30

Description

This is an umbrella ticket tracking the general effort to improve performance and interoperability between PySpark and Pandas. The core idea is to Apache Arrow as serialization format to reduce the overhead between PySpark and Pandas.

Issue Links

• incorporates: SPARK-21187 Complete support for remaining Spark data types in Arrow Converters

Sub-Tasks

• Refactor group aggregate pandas UDF to its own catalyst rules: Open
• Support GROUPED_AGG_PANDAS_UDF in Pivot: Open
• Clarify/Improve EvalType for grouped aggregate and window aggregate: Open