[TOREE-337] %AddPythonDeps magic to install packages from Pypi

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Labels: python, python-wheel
• Assignee: Unassigned
• Reporter: gsemet
• Created: 07/Sep/16 13:11
• Updated: 06/Oct/16 19:15

Description

I would like to volunteer to work on add two "magic" to Toree, related to Python dependency managements:

%AddPythonDeps
Usage: %AddPythonDeps pypi_package_name [version]

Download and install Python dependency from pypi.python.org and install it on the Spark cluster using pip. Transitive dependencies will be automatically retrieved.

—

%AddPythonDist
Usage: %AddPythonDist<url to dist>

Download the distribution package (source distribution package or binary distribution package or wheel) and install it on the master and workers using pip.

—

Example:
One would be able to specify something like

%AddPythonDeps numpy 1.1.1
%AddPythonDeps requests 0.1.1

I am working to bring pip and virtualenv support to Spark since we need to work on clean virtualenv. Here is a proposal on this subject and the pull request.

Let me know if it would make sense within Apache Toree to have such feature (I think it's cool to let each job describes its python dependencies and have Spark automatically handle them properly!)

Issue Links

• is related to: ZEPPELIN-1419 PySpark dependencies support
• relates to: SPARK-16367 Wheelhouse Support for PySpark