[SPARK-26355] Add a workaround for PyArrow 0.11.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 2.4.1, 3.0.0
• Component/s: PySpark, SQL
• Assignee: Takuya Ueshin
• Reporter: Takuya Ueshin
• Created: 13/Dec/18 03:55
• Updated: 12/Dec/22 18:10
• Resolved: 13/Dec/18 05:28

Description

In PyArrow 0.11, there is a API breaking change.

• ARROW-1949 - [Python/C++] Add option to Array.from_pandas and pyarrow.array to perform unsafe casts.

We should add a workaround to support PyArrow 0.11.

Issue Links

• links to: GitHub Pull Request #23305