Uploaded image for project: 'Livy'
  1. Livy
  2. LIVY-325

Refactoring Livy Session and Interpreter

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 0.4.0
    • Fix Version/s: None
    • Component/s: REPL
    • Labels:
      None

      Description

      Currently in Livy master code, Livy interpreter is bound with Livy session, when we created a session, we should specify which interpreter we want, and this interpreter will be created implicitly. This potentially has several limitations:

      1. We cannot create a share session, when we choose one language, we have to create a new session to use. But some notebooks like Zeppelin could use python, scala, R to manipulate data under the same SparkContext. So in Livy we should decouple interpreter with SC and support shared context between different interpreters.
      2. Furthermore, we cannot create multiple same interpreters in one session. For example in Zeppelin scope mode, it could create multiple scala interpreters to share with one context, but unfortunately in current Livy we could not support this.

      So based on the problems we mentioned above, we mainly have three things:

      1. Decouple interpreters from Spark context, so that we could create multiple interpreters under one context.
      2. Make sure multiple interpreters could be worked together.
      3. Change REST APIs to support multiple interpreters per session.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                jerryshao Saisai Shao
              • Votes:
                1 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated: