[SPARK-30834] Add note for recommended versions of Pandas and PyArrow for 2.4.x

• Type: Documentation
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.5
• Fix Version/s: 2.4.6, 3.0.0
• Component/s: Documentation, PySpark
• Assignee: Bryan Cutler
• Reporter: Bryan Cutler
• Created: 14/Feb/20 20:20
• Updated: 12/Dec/22 18:11
• Resolved: 17/Feb/20 02:10

Description

CI testing for branch 2.4 has been with the versions below. These are recommened and any newer versions can not be guaranteed correct.

 

for 2.4, python 3.6.8:

-bash-4.1$ python -c "import pandas; import pyarrow; print('pandas: %s' % pandas._version); print('pyarrow: %s' % pyarrow.version_)"
pandas: 0.19.2
pyarrow: 0.8.0

for master/3.0, python 3.6.8:

-bash-4.1$ python -c "import pandas; import pyarrow; print('pandas: %s' % pandas._version); print('pyarrow: %s' % pyarrow.version_)"
pandas: 0.24.2
pyarrow: 0.15.1

Issue Links

• links to: GitHub Pull Request #27586
• links to: GitHub Pull Request #27587