Description
Currently in Livy master code, Livy interpreter is bound with Livy session, when we created a session, we should specify which interpreter we want, and this interpreter will be created implicitly. This potentially has several limitations:
1. We cannot create a share session, when we choose one language, we have to create a new session to use. But some notebooks like Zeppelin could use python, scala, R to manipulate data under the same SparkContext. So in Livy we should decouple interpreter with SC and support shared context between different interpreters.
2. Furthermore, we cannot create multiple same interpreters in one session. For example in Zeppelin scope mode, it could create multiple scala interpreters to share with one context, but unfortunately in current Livy we could not support this.
So based on the problems we mentioned above, we mainly have three things:
1. Decouple interpreters from Spark context, so that we could create multiple interpreters under one context.
2. Make sure multiple interpreters could be worked together.
3. Change REST APIs to support multiple interpreters per session.
Attachments
Issue Links
- is related to
-
LIVY-666 Support named interpreter groups
-
- Open
-