[SPARK-13587] Support virtualenv in PySpark

• Type: New Feature
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.6.3, 2.0.2, 2.1.2, 2.2.1, 2.3.0
• Component/s: PySpark
• Assignee: Unassigned
• Reporter: Jeff Zhang
• Created: 01/Mar/16 04:57
• Updated: 25/Apr/21 11:25

Description

Currently, it's not easy for user to add third party python packages in pyspark.

• One way is to using --py-files (suitable for simple dependency, but not suitable for complicated dependency, especially with transitive dependency)
• Another way is install packages manually on each node (time wasting, and not easy to switch to different environment)

Python has now 2 different virtualenv implementation. One is native virtualenv another is through conda. This jira is trying to migrate these 2 tools to distributed environment

Issue Links

• is depended upon by: ZEPPELIN-2233 Virtualenv support for PySparkInterpreter
• is duplicated by: SPARK-19095 virtualenv example does not work in yarn cluster mode
• is duplicated by: SPARK-16367 Wheelhouse Support for PySpark
• is related to: SPARK-16367 Wheelhouse Support for PySpark
• is superceded by: SPARK-20001 Support PythonRunner executing inside a Conda env
• relates to: SPARK-17428 SparkR executors/workers support virtualenv
• relates to: SPARK-6764 Add wheel package support for PySpark
• links to: [Github] Pull Request #13599 (zjffdu)
• links to: GitHub Pull Request #13599
• links to: Virtualenv Doc

Sub-Tasks

• virtualenv example does not work in yarn cluster mode: Resolved
• Kmeans.py application fails with virtualenv and due to parse error: Resolved
• virtualenv example failed with conda due to ImportError: No module named ruamel.yaml.comments: Resolved