Attach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • None
    • None

    Description

      Helium proposal https://cwiki.apache.org/confluence/display/ZEPPELIN/Helium+proposal was about pluggable application on top of Zeppelin. I'd like to extend the concept and make Helium covers interpreters, notebook storages, too.
      i.e. Interpreter and notebook storage are able to loaded from maven repository on the fly.

      Interpreter

      Interpreter is already pluggable. User can just drop necessary binary under interpreter/[interpretername], add configuration into conf/zeppelin-site.xml and restart Zeppelin will make interpreter work.
      But it'll be much easier if Zeppelin can load interpreter binary from maven repository and load them without restart.

      Notebook storage

      Notebook storage is also already pluggable. User can drop necessary binary into ZeppelinServer's classpath (/lib), add configuraion into conf/zeppelin-site.xml and restart Zeppelin will make notebook storage work.
      Current implementation may bring library conflict while all notebook storage is loaded from the same classloader. There should be dedicated classloader per implementation like Interpreter. And then it can be loaded from maven repository on the fly, just like an Interpreter.

      Application

      https://cwiki.apache.org/confluence/display/ZEPPELIN/Helium+proposal tried to define it. Application can be loaded and launched just like Interpreter and Notebook storage, but not in ZeppelinServer process but in interpreter process because of some application may want to use Interpreter provided resource (e.g. SparkContext). The resource will be acceseed through Resource pool. Once application is being loaded, application will able to use outputstream provided by Zeppelin as well as InterpreterContext (provides api for angularobject, etc).
      In short, not only the code provided by user but also the packaged code will be run on the notebook.

      Package registry

      Package registry central repository for package metadata. Package metadata is informations such as name, type, description, maven repository, version, etc. One simplest way of making this package repository is create an separate git repository and use it as a package repository. We will need automated registration / update process of metadata from 3rd party developers.

      • 'Helium' menu*
        'Helium' menu on GUI will get metadata from package registry and list them to user select. It'll need to save user's selection into conf/helium.json

      Following sub tasks can be created

      • [Interpreter] Make interpreter loadable from maven repository and REST api for it
      • [Notebook storage] Make each notebook storage uses separate classloader
      • [Notebook storage] Make notebook storage loadable from maven repository and REST api for it
      • [Application] Provide output stream to application
      • [Application] Shared resource pool that user application can exchange data between interpreter processes
      • [Application] Make each user application loaded from separate classloader
      • [Application] Add 'paragraph' scope for angular object in addition to 'notebook', 'global'
      • [Application] Application launch API
      • [GUI] Create Helium menu and provide user interface for REST api
      • [GUI] Display package from package registry
      • [GUI] Save / Load user selection into conf/helium.conf
      • [Registry] Helium package registry

      Attachments

        Issue Links

        1.
        Load interpreter from maven repository Sub-task Resolved Mina Lee   Actions
        2.
        Add 'paragraph' scope for angular object Sub-task Resolved Lee Moon Soo   Actions
        3.
        Hyperlink to npm package page Sub-task Resolved Unassigned   Actions
        4.
        Provide OutputStream from InterpreterContext Sub-task Resolved Lee Moon Soo   Actions
        5.
        Shared resource pool that user application can exchange data across interpreter processes Sub-task Resolved Lee Moon Soo   Actions
        6.
        Release resource automatically from resource pool when paragraph / notebook deletion from ResourcePool Sub-task Resolved Lee Moon Soo   Actions
        7.
        Packaged user code that runs on interpreter process Sub-task Resolved Lee Moon Soo

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 20m
        Actions
        8.
        Define abstract API for visualization and refactor build-in visualizations Sub-task Resolved Lee Moon Soo   Actions
        9.
        Create / manage settings inside of each visualization. Sub-task Resolved Lee Moon Soo   Actions
        10.
        Let helium implement visualization API Sub-task Resolved Lee Moon Soo   Actions
        11.
        Modernize visualization/transformation using ES6 class Sub-task Resolved Lee Moon Soo   Actions
        12.
        List all available Helium packages under zeppelin.apache.org Sub-task Resolved Ahyoung Ryu   Actions
        13.
        List helium packages in Zeppelin GUI by reading file provided by ZEPPELIN-1973 Sub-task Resolved Lee Moon Soo   Actions
        14.
        Need to mention that user needs to add "icon" field to package.json Sub-task Open Ahyoung Ryu   Actions
        15.
        Need to make Helium packages sortable in Helium menu Sub-task Resolved Ahyoung Ryu   Actions
        16.
        Need a pagination for Helium packages Sub-task Resolved Ahyoung Ryu   Actions
        17.
        Need search bar in Helium menu Sub-task Open Ahyoung Ryu   Actions

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            moon Lee Moon Soo

            Dates

              Created:
              Updated:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 40m
                40m

                Slack

                  Issue deployment