[LIVY-106] Support multiple versions of Spark

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.2
• Fix Version/s: 0.9.0
• Component/s: Core
• Assignee: Unassigned
• Reporter: Marcelo Masiero Vanzin
• Created: 04/Apr/16 18:14
• Updated: 12/Nov/22 22:10

Description

It would be nice to support multiple versions of Spark; the Livy server doesn't have a direct dependency on Spark, so users could just say which Spark version they want when launching sessions.

Livy would need some mapping between versions and different SPARK_HOME values, along with potentially different configuration files / directories too.

Issue Links

• is a parent of: LIVY-246 Support multiple Spark home in runtime
• is a parent of: LIVY-162 Support Spark 2
• is a parent of: LIVY-219 Change Livy IT to support different scala version of livy-repl
• is a parent of: LIVY-227 Propose to ship two REPL bundles in one assembly
• is a parent of: LIVY-228 Shade some repl dependencies to mitigate the building difference between Spark1 and 2
• is a parent of: LIVY-233 Support SparkSession in Job API
• is a parent of: LIVY-234 Make Livy Scala API to support both Scala 2.10 and 2.11