[SPARK-29376] Upgrade Apache Arrow to 0.15.1

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: PySpark, SQL
• Assignee: Unassigned
• Reporter: Bryan Cutler
• Created: 07/Oct/19 18:47
• Updated: 12/Dec/22 18:10
• Resolved: 15/Nov/19 04:28

Description

Apache Arrow 0.15.0 was just released see https://arrow.apache.org/blog/2019/10/06/0.15.0-release/

There are a number of fixes and improvements including a change to the binary IPC format https://issues.apache.org/jira/browse/ARROW-6313.

The next planned release will be 1.0.0, so it would be good to upgrade Spark as a preliminary step.

Updated to use Apache Arrow 0.15.1, which was released soon after 0.15.0 to include some important bug fixes.

change log at https://arrow.apache.org/release/0.15.1.html

Issue Links

• is depended upon by: SPARK-29493 Add MapType support for Arrow Java
• is related to: SPARK-26566 Upgrade apache/arrow to 0.12.0
• is related to: SPARK-29378 Upgrade SparkR to use Arrow 0.15 API
• relates to: SPARK-32312 Upgrade Apache Arrow to 1.0.0
• links to: GitHub Pull Request #26133