[SPARK-5929] Pyspark: Register a pip requirements file with spark_context

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Won't Fix
• Component/s: PySpark
• Assignee: Unassigned
• Reporter: buckhx
• Created: 20/Feb/15 21:03
• Updated: 17/Feb/17 09:15
• Resolved: 17/Feb/17 09:15

Description

I've been doing a lot of dependency work with shipping dependencies to workers as it is non-trivial for me to have my workers include the proper dependencies in their own environments.

To circumvent this, I added a addRequirementsFile() method that takes a pip requirements file, downloads the packages, repackages them to be registered with addPyFiles and ship them to workers.

Here is a comparison of what I've done on the Palantir fork

https://github.com/buckheroux/spark/compare/palantir:master...master

Issue Links

• links to: [Github] Pull Request #4743 (buckheroux)
• links to: [Github] Pull Request #4897 (buckheroux)
• links to: [Github] Pull Request #12398 (buckhx)