[ARROW-288] Implement Arrow adapter for Spark Datasets

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.5.0
• Component/s: C++, Java
• External issue URL: https://github.com/apache/arrow/issues/15668
• Assignee: Bryan Cutler
• Reporter: Wes McKinney
• Created: 11/Sep/16 17:02
• Updated: 11/Jan/23 07:07
• Resolved: 23/Aug/17 14:32

Description

It would be valuable for applications that use Arrow to be able to

• Convert between Spark DataFrames/Datasets and Java Arrow vectors
• Send / Receive Arrow record batches / Arrow file format RPCs to / from Spark
• Allow PySpark to use Arrow for messaging in UDF evaluation

Issue Links

• relates to: SPARK-13534 Implement Apache Arrow serializer for Spark DataFrame for use in DataFrame.toPandas