[BEAM-4032] Support staging binary distributions of dependency packages.

• Type: Improvement
• Status: Triage Needed
• Priority: P2
• Resolution: Fixed
• Fix Version/s: 2.38.0
• Component/s: sdk-py-core
• Assignee: Anand Inguva
• Reporter: Valentyn Tymofieiev
• Created: 09/Apr/18 17:24
• Updated: 13/Apr/23 11:09
• Resolved: 12/Feb/22 05:48

Description

requirements.txt only supports source-distribution dependencies [1].

--extra_packages does not officially support wheel files [2].

It is possible to expand this to support binary distributions as long as we have the knowledge of the target platform.

We should take into consideration the mechanisms of staging dependencies through portability framework, and perhaps consolidate some of the existing options.

https://github.com/apache/beam/blob/a79d1b4fc27eb81db0d9a773047820a206f3d238/sdks/python/apache_beam/runners/dataflow/internal/dependency.py#L260

https://github.com/apache/beam/blob/a79d1b4fc27eb81db0d9a773047820a206f3d238/sdks/python/apache_beam/runners/dataflow/internal/dependency.py#L188

 

Issue Links

• is related to: BEAM-13719 Install dependencies from requirements.txt on the container image when pre-building container image
• relates to: BEAM-14176 Beam dataflow hangs with requirements.txt
• relates to: BEAM-13314 Revise recommendations to manage Python pipeline dependencies.
• links to: Documentation for staging binaries
• links to: GitHub Pull Request #16448
• links to: GitHub Pull Request #16633