[SPARK-23874] Upgrade apache/arrow to 0.10.0

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3.0
• Fix Version/s: 2.4.0
• Component/s: PySpark
• Target Version/s: 2.4.0
• Assignee: Bryan Cutler
• Reporter: Xiao Li
• Created: 05/Apr/18 04:52
• Updated: 12/Dec/22 18:10
• Resolved: 15/Aug/18 00:14

Description

Version 0.10.0 will allow for the following improvements and bug fixes:

• Allow for adding BinaryType support ARROW-2141
• Bug fix related to array serialization ARROW-1973
• Python2 str will be made into an Arrow string instead of bytes ARROW-2101
• Python bytearrays are supported in as input to pyarrow ARROW-2141
• Java has common interface for reset to cleanup complex vectors in Spark ArrowWriter ARROW-1962
• Cleanup pyarrow type equality checks ARROW-2423
• ArrowStreamWriter should not hold references to ArrowBlocks ARROW-2632, ARROW-2645
• Improved low level handling of messages for RecordBatch ARROW-2704

 

 

Issue Links

• blocks: SPARK-23030 Decrease memory consumption with toPandas() collection using Arrow
• blocks: SPARK-23555 Add BinaryType support for Arrow in PySpark
• is blocked by: ARROW-2432 [Python] from_pandas fails when converting decimals if have None values
• is cloned by: SPARK-26566 Upgrade apache/arrow to 0.12.0
• links to: [Github] Pull Request #21013 (BryanCutler)
• links to: [Github] Pull Request #21939 (BryanCutler)