Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-8918

Add Streaming Expressions to the admin page

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 6.1, 7.0
    • Component/s: Admin UI
    • Labels:
      None

      Description

      Add to the admin page an ability to work with and view Streaming Expressions.

      This tab will appear under the Collection selection section and will work similarly to the Query tab. On this page the user will be able to enter a streaming expression for execution. The user can then execute the expression against the collection and view all the results as they are returned from the Stream handler. Along with this the user will be able to view the structure of the expression in a graph-like layout.

      If the user wishes to only view the expression structure without executing the expression the user will be able to click an "Explain" button which will show the structure. Included in the structure will be information about each node (the expression for that node).

      1. sample-display.png
        133 kB
        Dennis Gove
      2. sample-display.png
        116 kB
        Dennis Gove
      3. SOLR-8918.patch
        18 kB
        Dennis Gove
      4. SOLR-8918.patch
        18 kB
        Dennis Gove
      5. SOLR-8918.patch
        142 kB
        Dennis Gove
      6. SOLR-8918.patch
        144 kB
        Dennis Gove
      7. SOLR-8918.patch
        31 kB
        Dennis Gove

        Issue Links

          Activity

          Hide
          dpgove Dennis Gove added a comment - - edited

          This patch adds a new section to work with Streaming Expressions. It is far from complete but I wanted to put it up to get people's feedback or ideas about the display.

          Executing an expression should show all the results. Explaining an expression is still under development but if you click that button it will show you the layout of a sample expression. Still requires work on the layout, canvas size, node display (I'm thinking different types of rectangles w/images for different types of Stream classes), and getting the expression explanation from the stream handler.

          Show
          dpgove Dennis Gove added a comment - - edited This patch adds a new section to work with Streaming Expressions. It is far from complete but I wanted to put it up to get people's feedback or ideas about the display. Executing an expression should show all the results. Explaining an expression is still under development but if you click that button it will show you the layout of a sample expression. Still requires work on the layout, canvas size, node display (I'm thinking different types of rectangles w/images for different types of Stream classes), and getting the expression explanation from the stream handler.
          Hide
          dpgove Dennis Gove added a comment -

          Attaching a sample display (without search results)

          Show
          dpgove Dennis Gove added a comment - Attaching a sample display (without search results)
          Hide
          dpgove Dennis Gove added a comment -

          Includes changes in the solrj.io package to support returning a stream explanation as well as some changes on the UI to display what is returned. Still requires a bit of work for UI and some additional testing (both manual and unit) for the stream explanations.

          Show
          dpgove Dennis Gove added a comment - Includes changes in the solrj.io package to support returning a stream explanation as well as some changes on the UI to display what is returned. Still requires a bit of work for UI and some additional testing (both manual and unit) for the stream explanations.
          Hide
          upayavira Upayavira added a comment -

          This is great! Will really help folks learn the streaming API.

          Some questions:

          Would it make sense to add two tabs at the top, one for streaming expressions, and the other for SQL?

          The second suggestion is a lot more substantial. Given that the streaming API is essentially a programming language, would it be possible to add support for autocomplete or code suggestions to the editor?

          There's a list of Javascript editors here, some of which support "code suggestions":
          https://en.wikipedia.org/wiki/Comparison_of_JavaScript-based_source_code_editors

          Thoughts?

          Show
          upayavira Upayavira added a comment - This is great! Will really help folks learn the streaming API. Some questions: Would it make sense to add two tabs at the top, one for streaming expressions, and the other for SQL? The second suggestion is a lot more substantial. Given that the streaming API is essentially a programming language, would it be possible to add support for autocomplete or code suggestions to the editor? There's a list of Javascript editors here, some of which support "code suggestions": https://en.wikipedia.org/wiki/Comparison_of_JavaScript-based_source_code_editors Thoughts?
          Hide
          dpgove Dennis Gove added a comment -

          UI changes to display the explanation nodes properly as a graph. Support for additional streams.

          Still requires addition of tests.

          Show
          dpgove Dennis Gove added a comment - UI changes to display the explanation nodes properly as a graph. Support for additional streams. Still requires addition of tests.
          Hide
          dpgove Dennis Gove added a comment - - edited

          Right now you can get additional details on each node (expression, stream type, implementing class) if you hover-over. I'd like to add the ability to click the node and have the details appear somewhere on the page. Also would like to change the node color based on the type of stream (source, decorator, graph, etc...) and add a legend for that.

          Show
          dpgove Dennis Gove added a comment - - edited Right now you can get additional details on each node (expression, stream type, implementing class) if you hover-over. I'd like to add the ability to click the node and have the details appear somewhere on the page. Also would like to change the node color based on the type of stream (source, decorator, graph, etc...) and add a legend for that.
          Hide
          dpgove Dennis Gove added a comment -

          Have finally got this in a finalized state. The admin page allows you to select whether you want an explanation with your results or just the results. The explanation has four types of nodes: Stream Decorator, Stream Source, Graph Source, and Datastore. Each type shows in a different color. For those with color-blindness the type of stream is also available in the node hover. I've also added a legend. I did not make the nodes clickable - am leaving that for future work.

          I'll be splitting this up into two tickets. This ticket will cover the admin page changes and a new ticket will be added covering the changes on streaming classes to return the explanation.

          Show
          dpgove Dennis Gove added a comment - Have finally got this in a finalized state. The admin page allows you to select whether you want an explanation with your results or just the results. The explanation has four types of nodes: Stream Decorator, Stream Source, Graph Source, and Datastore. Each type shows in a different color. For those with color-blindness the type of stream is also available in the node hover. I've also added a legend. I did not make the nodes clickable - am leaving that for future work. I'll be splitting this up into two tickets. This ticket will cover the admin page changes and a new ticket will be added covering the changes on streaming classes to return the explanation.
          Hide
          dpgove Dennis Gove added a comment -

          Patch for admin page changes only.

          Show
          dpgove Dennis Gove added a comment - Patch for admin page changes only.
          Hide
          dpgove Dennis Gove added a comment -

          Changes the format of the output resultset data to better show json.

          Show
          dpgove Dennis Gove added a comment - Changes the format of the output resultset data to better show json.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit af7dad6825d47e76c39842e97be8b37ab4c2cffd in lucene-solr's branch refs/heads/master from Dennis Gove
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=af7dad6 ]

          SOLR-8918: Adds Streaming to the admin page under the collections section

          Includes ability to see graphically the expression explanation

          Show
          jira-bot ASF subversion and git services added a comment - Commit af7dad6825d47e76c39842e97be8b37ab4c2cffd in lucene-solr's branch refs/heads/master from Dennis Gove [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=af7dad6 ] SOLR-8918 : Adds Streaming to the admin page under the collections section Includes ability to see graphically the expression explanation
          Hide
          joel.bernstein Joel Bernstein added a comment -

          Playing with this now. This is an amazing piece of work...

          Show
          joel.bernstein Joel Bernstein added a comment - Playing with this now. This is an amazing piece of work...
          Hide
          joel.bernstein Joel Bernstein added a comment -

          One thing I noticed is the "Graph" under the Cloud tab has disappeared. Probably related to this ticket.

          Show
          joel.bernstein Joel Bernstein added a comment - One thing I noticed is the "Graph" under the Cloud tab has disappeared. Probably related to this ticket.
          Hide
          dpgove Dennis Gove added a comment -

          I've confirmed that it's definitely this patch that caused the graph on the cloud tab to disappear but I've yet to figure out why it's disappeared. I suspect angular is doing something I didn't expect, particularly with the directive at https://github.com/apache/lucene-solr/blob/master/solr/webapp/web/js/angular/controllers/stream.js#L146

          Am continuing to investigate.

          Show
          dpgove Dennis Gove added a comment - I've confirmed that it's definitely this patch that caused the graph on the cloud tab to disappear but I've yet to figure out why it's disappeared. I suspect angular is doing something I didn't expect, particularly with the directive at https://github.com/apache/lucene-solr/blob/master/solr/webapp/web/js/angular/controllers/stream.js#L146 Am continuing to investigate.
          Hide
          dpgove Dennis Gove added a comment -

          I've found the problem and in doing so developed a slight dislike for angular (but really it was my own fault because I know all about javascript global states and what not).

          stream.js was created from a copy of cloud.js. cloud.js created a global variable called graphSubController. When I was mucking around with stream.js I kept the graphSubController global variable but re-implemented it. Now, when angular runs it will process all the .js files and in doing so the stream.js::graphSubController replaced cloud.js::graphSubController so the correct sub controller was not being called from the cloud.js code page.

          I'll have a patch to fix this in a little bit - after I go and cleanup all the alerts I added to figure out what in the world was going on.

          (in reality though, it was pretty fun to learn-ish angular for this)

          Show
          dpgove Dennis Gove added a comment - I've found the problem and in doing so developed a slight dislike for angular (but really it was my own fault because I know all about javascript global states and what not). stream.js was created from a copy of cloud.js. cloud.js created a global variable called graphSubController. When I was mucking around with stream.js I kept the graphSubController global variable but re-implemented it. Now, when angular runs it will process all the .js files and in doing so the stream.js::graphSubController replaced cloud.js::graphSubController so the correct sub controller was not being called from the cloud.js code page. I'll have a patch to fix this in a little bit - after I go and cleanup all the alerts I added to figure out what in the world was going on. (in reality though, it was pretty fun to learn-ish angular for this)
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit e6e495c79588c60db1ac45bcba1a1dcaa970bcea in lucene-solr's branch refs/heads/master from Dennis Gove
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=e6e495c ]

          SOLR-8918: Corrects usage of a global variable in admin page's stream.js which was overriding the same variable in cloud.js

          Show
          jira-bot ASF subversion and git services added a comment - Commit e6e495c79588c60db1ac45bcba1a1dcaa970bcea in lucene-solr's branch refs/heads/master from Dennis Gove [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=e6e495c ] SOLR-8918 : Corrects usage of a global variable in admin page's stream.js which was overriding the same variable in cloud.js
          Hide
          upayavira Upayavira added a comment -

          You are right, globals suck. That's js not angular. You could add the subcontroller to the scope, that'd make it local.

          Show
          upayavira Upayavira added a comment - You are right, globals suck. That's js not angular. You could add the subcontroller to the scope, that'd make it local.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit fc2200e4f9927e5595197831b9b6265085b65ab3 in lucene-solr's branch refs/heads/branch_6x from Dennis Gove
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=fc2200e ]

          SOLR-8918: Adds Streaming to the admin page under the collections section

          Includes ability to see graphically the expression explanation

          Show
          jira-bot ASF subversion and git services added a comment - Commit fc2200e4f9927e5595197831b9b6265085b65ab3 in lucene-solr's branch refs/heads/branch_6x from Dennis Gove [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=fc2200e ] SOLR-8918 : Adds Streaming to the admin page under the collections section Includes ability to see graphically the expression explanation
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 499ae951ec4a5e35a9d067f2d3abf69e6734a08b in lucene-solr's branch refs/heads/branch_6x from Dennis Gove
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=499ae95 ]

          SOLR-8918: Corrects usage of a global variable in admin page's stream.js which was overriding the same variable in cloud.js

          Show
          jira-bot ASF subversion and git services added a comment - Commit 499ae951ec4a5e35a9d067f2d3abf69e6734a08b in lucene-solr's branch refs/heads/branch_6x from Dennis Gove [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=499ae95 ] SOLR-8918 : Corrects usage of a global variable in admin page's stream.js which was overriding the same variable in cloud.js
          Hide
          hossman Hoss Man added a comment -

          Manually correcting fixVersion per Step #S5 of LUCENE-7271

          Show
          hossman Hoss Man added a comment - Manually correcting fixVersion per Step #S5 of LUCENE-7271

            People

            • Assignee:
              dpgove Dennis Gove
              Reporter:
              dpgove Dennis Gove
            • Votes:
              1 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development