[SPARK-45143] Make PySpark compatible with PyArrow 13.0.0

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: Connect, PySpark
• Labels: pull-request-available
• Assignee: Ruifeng Zheng
• Reporter: Ruifeng Zheng
• Created: 13/Sep/23 05:18
• Updated: 17/Mar/24 21:55
• Resolved: 15/Sep/23 04:26

Description

https://github.com/apache/spark/actions/runs/6167186123/job/16738683872

 

======================================================================
FAIL [0.095s]: test_from_to_pandas (pyspark.pandas.tests.data_type_ops.test_datetime_ops.DatetimeOpsTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/__w/spark/spark/python/pyspark/testing/pandasutils.py", line 122, in _assert_pandas_equal
    assert_series_equal(
  File "/usr/local/lib/python3.9/dist-packages/pandas/_testing/asserters.py", line 931, in assert_series_equal
    assert_attr_equal("dtype", left, right, obj=f"Attributes of {obj}")
  File "/usr/local/lib/python3.9/dist-packages/pandas/_testing/asserters.py", line 415, in assert_attr_equal
    raise_assert_detail(obj, msg, left_attr, right_attr)
  File "/usr/local/lib/python3.9/dist-packages/pandas/_testing/asserters.py", line 599, in raise_assert_detail
    raise AssertionError(msg)
AssertionError: Attributes of Series are different

Attribute "dtype" are different
[left]:  datetime64[ns]
[right]: datetime64[us] 

Issue Links

• is related to: SPARK-45141 Pin `pyarrow==12.0.1` in CI
• is related to: SPARK-47432 Add `pyarrow` upper bound requirement, `<13.0.0`
• links to: GitHub Pull Request #42920