[BEAM-8900] Python SDK should stage only up-to-date versions of pipeline dependencies defined by requirements file.

• Type: Bug
• Status: Open
• Priority: P3
• Resolution: Unresolved
• Component/s: sdk-py-core
• Labels: starter
• Assignee: Unassigned
• Reporter: Valentyn Tymofieiev
• Created: 05/Dec/19 23:21
• Updated: 04/Jun/22 14:37

Description

When staging pipeline dependencies specified in requirements.txt file and transitive deps, we stage entire content of requirements-cache directory on user's machine, which may include dependencies that were cached a while ago and are no longer needed. See [1] for discussion and some ideas how we can avoid staging packages we don't need.

[1] https://lists.apache.org/thread.html/f99820489026a46160505ea9a4c8c4a0eabe8649143c1ee22b7088aa%40%3Cdev.beam.apache.org%3E