[ZEPPELIN-18] Running pyspark without deploying python libraries to every yarn node

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.5.0
• Assignee: Jongyoul Lee
• Reporter: Jongyoul Lee
• Created: 01/Apr/15 01:01
• Updated: 05/Jul/15 17:49
• Resolved: 05/Jul/15 17:49

Description

We only use python sources under SPARK_HOME/python for PySpark. I think copying these files into /python is better idea because it doesn't need to extract - or build - spark no longer. If PySpark has changed by upgrading Spark version, It's easy to follow up by coping new python files into zeppelin. If we should support multiple version of spark, we can solve it of having multiple python files.

Issue Links

• is related to: SPARK-6869 Add pyspark archives path to PYTHONPATH