[BEAM-6765] Beam 2.10.0 for Python requires pyarrow 0.11.1, which is not installable in Google Cloud DataFlow

• Type: Bug
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Affects Version/s: 2.10.0
• Fix Version/s: 2.10.0
• Component/s: sdk-py-core
• Assignee: Unassigned
• Reporter: Barry Hart
• Created: 04/Mar/19 13:50
• Updated: 24/Jul/20 20:01
• Resolved: 14/Mar/19 17:54

Description

When trying to run a Beam 2.10.0 job in Google Cloud DataFlow, I get the following error:

Collecting pyarrow==0.11.1 (from -r requirements.txt (line 51))
Could not find a version that satisfies the requirement pyarrow==0.11.1 (from -r requirements.txt (line 51)) (from versions: 0.9.0, 0.10.0, 0.11.0, 0.12.1)
No matching distribution found for pyarrow==0.11.1 (from -r requirements.txt (line 51))

This version, while it exists, cannot be installed in Google Cloud DataFlow, because it is only available on PyPI as a wheel, and DataFlow does not allow installing binary packages, only source packages.