Details
-
New Feature
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
-
None
Description
Helium proposal https://cwiki.apache.org/confluence/display/ZEPPELIN/Helium+proposal was about pluggable application on top of Zeppelin. I'd like to extend the concept and make Helium covers interpreters, notebook storages, too.
i.e. Interpreter and notebook storage are able to loaded from maven repository on the fly.
Interpreter
Interpreter is already pluggable. User can just drop necessary binary under interpreter/[interpretername], add configuration into conf/zeppelin-site.xml and restart Zeppelin will make interpreter work.
But it'll be much easier if Zeppelin can load interpreter binary from maven repository and load them without restart.
Notebook storage
Notebook storage is also already pluggable. User can drop necessary binary into ZeppelinServer's classpath (/lib), add configuraion into conf/zeppelin-site.xml and restart Zeppelin will make notebook storage work.
Current implementation may bring library conflict while all notebook storage is loaded from the same classloader. There should be dedicated classloader per implementation like Interpreter. And then it can be loaded from maven repository on the fly, just like an Interpreter.
Application
https://cwiki.apache.org/confluence/display/ZEPPELIN/Helium+proposal tried to define it. Application can be loaded and launched just like Interpreter and Notebook storage, but not in ZeppelinServer process but in interpreter process because of some application may want to use Interpreter provided resource (e.g. SparkContext). The resource will be acceseed through Resource pool. Once application is being loaded, application will able to use outputstream provided by Zeppelin as well as InterpreterContext (provides api for angularobject, etc).
In short, not only the code provided by user but also the packaged code will be run on the notebook.
Package registry
Package registry central repository for package metadata. Package metadata is informations such as name, type, description, maven repository, version, etc. One simplest way of making this package repository is create an separate git repository and use it as a package repository. We will need automated registration / update process of metadata from 3rd party developers.
- 'Helium' menu*
'Helium' menu on GUI will get metadata from package registry and list them to user select. It'll need to save user's selection into conf/helium.json
Following sub tasks can be created
- [Interpreter] Make interpreter loadable from maven repository and REST api for it
- [Notebook storage] Make each notebook storage uses separate classloader
- [Notebook storage] Make notebook storage loadable from maven repository and REST api for it
- [Application] Provide output stream to application
- [Application] Shared resource pool that user application can exchange data between interpreter processes
- [Application] Make each user application loaded from separate classloader
- [Application] Add 'paragraph' scope for angular object in addition to 'notebook', 'global'
- [Application] Application launch API
- [GUI] Create Helium menu and provide user interface for REST api
- [GUI] Display package from package registry
- [GUI] Save / Load user selection into conf/helium.conf
- [Registry] Helium package registry
