Uploaded image for project: 'Livy'
  1. Livy
  2. LIVY-325

Refactoring Livy Session and Interpreter

Attach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments


    • Type: New Feature
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 0.4.0
    • Fix Version/s: None
    • Component/s: REPL
    • Labels:


      Currently in Livy master code, Livy interpreter is bound with Livy session, when we created a session, we should specify which interpreter we want, and this interpreter will be created implicitly. This potentially has several limitations:

      1. We cannot create a share session, when we choose one language, we have to create a new session to use. But some notebooks like Zeppelin could use python, scala, R to manipulate data under the same SparkContext. So in Livy we should decouple interpreter with SC and support shared context between different interpreters.
      2. Furthermore, we cannot create multiple same interpreters in one session. For example in Zeppelin scope mode, it could create multiple scala interpreters to share with one context, but unfortunately in current Livy we could not support this.

      So based on the problems we mentioned above, we mainly have three things:

      1. Decouple interpreters from Spark context, so that we could create multiple interpreters under one context.
      2. Make sure multiple interpreters could be worked together.
      3. Change REST APIs to support multiple interpreters per session.


        Issue Links



            • Assignee:
              jerryshao Saisai Shao


              • Created:

                Issue deployment