Solr
  1. Solr
  2. SOLR-5810

State of external collections not displayed in cloud graph panel

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 5.0, 6.0
    • Component/s: SolrCloud, web gui
    • Labels:
      None

      Description

      External collections (SOLR-5473) are not displayed in the Cloud -> graph panel, which makes it very hard to see which external collections have problems, such as after a downed node comes back online.

      1. SOLR-5810.patch
        37 kB
        Timothy Potter
      2. SOLR-5810.patch
        35 kB
        Timothy Potter
      3. SOLR-5810.patch
        35 kB
        Timothy Potter
      4. SOLR-5810.prelim2.patch
        27 kB
        Timothy Potter
      5. SOLR-5810-prelim.patch
        4 kB
        Timothy Potter

        Issue Links

          Activity

          Hide
          Jessica Cheng Mallet added a comment -

          Please make sure that when this is implemented, it is done in a scalable way with many collections (e.g. pagination, etc.).

          Show
          Jessica Cheng Mallet added a comment - Please make sure that when this is implemented, it is done in a scalable way with many collections (e.g. pagination, etc.).
          Hide
          Noble Paul added a comment -

          we should change the UI to just expand one collection at a time (irrespective of it is external or internal)

          Show
          Noble Paul added a comment - we should change the UI to just expand one collection at a time (irrespective of it is external or internal)
          Hide
          Timothy Potter added a comment -

          I thinking we need paging with adjustable page sizes (default 20). With paging, there's an explicit or implicit sort order; might be nice to sort collections by creation date, num docs, activity level <-

          {might be too slow to get this for many collections}

          besides name (default ordering)

          It would also be nice to be able to filter by:
          + name prefix matching (tj*)
          + config name (to show me all the collections that use config xyz)
          + activity level (to show my busiest collections in the past X time range)
          + replica status (to show me all the collections that have replicas that are down | recovering | etc)
          + by node (to show me all the collections that have replicas on a specific node in my cluster)
          + creation date (to show me all the collections created since some date or before some other date)

          Show
          Timothy Potter added a comment - I thinking we need paging with adjustable page sizes (default 20). With paging, there's an explicit or implicit sort order; might be nice to sort collections by creation date, num docs, activity level <- {might be too slow to get this for many collections} besides name (default ordering) It would also be nice to be able to filter by: + name prefix matching (tj*) + config name (to show me all the collections that use config xyz) + activity level (to show my busiest collections in the past X time range) + replica status (to show me all the collections that have replicas that are down | recovering | etc) + by node (to show me all the collections that have replicas on a specific node in my cluster) + creation date (to show me all the collections created since some date or before some other date)
          Hide
          Timothy Potter added a comment -

          This is preliminary patch to allow me to visualize external collections for testing SOLR-5473. It doesn't implement any paging or filtering as we really need for handling a large number of external collections. I'm simply posting it here for others helping out get support for external collections hardened.

          Show
          Timothy Potter added a comment - This is preliminary patch to allow me to visualize external collections for testing SOLR-5473 . It doesn't implement any paging or filtering as we really need for handling a large number of external collections. I'm simply posting it here for others helping out get support for external collections hardened.
          Hide
          Noble Paul added a comment -

          The most important features as I see are

          • pattern matching search e.g: *coll*1 which is case insensitive. The names are not in millions so we can do an O search. An auto complete type textbox should be fine. IntelliJ class lookup is a good example
          • filter to show recovering/healthy/down collections

          I don't really mind having a scrolling frame with all the names sorted alphabetically , as long as I can reach to the collection easily with a search. A typical CRUD application is likely an overkill

          Show
          Noble Paul added a comment - The most important features as I see are pattern matching search e.g: *coll*1 which is case insensitive. The names are not in millions so we can do an O search. An auto complete type textbox should be fine. IntelliJ class lookup is a good example filter to show recovering/healthy/down collections I don't really mind having a scrolling frame with all the names sorted alphabetically , as long as I can reach to the collection easily with a search. A typical CRUD application is likely an overkill
          Hide
          Timothy Potter added a comment -

          Still a work in progress, but basic paging support is in place with the option to apply a simple filter on the collection name (see screen shot for UI concept). The code in this patch depends on the code for SOLR-5473, so is not ready for general consumption yet. Posting now to get some feedback on the approach and UI code; server-side code should be close (except it needs a unit test).

          Show
          Timothy Potter added a comment - Still a work in progress, but basic paging support is in place with the option to apply a simple filter on the collection name (see screen shot for UI concept). The code in this patch depends on the code for SOLR-5473 , so is not ready for general consumption yet. Posting now to get some feedback on the approach and UI code; server-side code should be close (except it needs a unit test).
          Hide
          Timothy Potter added a comment -

          can't get screen shot tool to work

          Show
          Timothy Potter added a comment - can't get screen shot tool to work
          Hide
          Mark Miller added a comment -

          How is this going Timothy Potter? This seems like a critical part of SOLR-5473 , so I'd like to get them in together.

          Show
          Mark Miller added a comment - How is this going Timothy Potter ? This seems like a critical part of SOLR-5473 , so I'd like to get them in together.
          Hide
          Noble Paul added a comment -

          There are multiple parts to this ticket.

          1. Get all collections in the cloud panel
          2. Display collections in pages rather than one large single page
          3. And a way to 'search' for a particular collection

          #1 is a must for 5473 . I have tested the other two and they are working . Just not sure if the latest patch is attached with this jira

          Show
          Noble Paul added a comment - There are multiple parts to this ticket. Get all collections in the cloud panel Display collections in pages rather than one large single page And a way to 'search' for a particular collection #1 is a must for 5473 . I have tested the other two and they are working . Just not sure if the latest patch is attached with this jira
          Hide
          Timothy Potter added a comment -

          I need to create an updated patch out of our internal repo where we originally developed this solution. I can get to this later today.

          Show
          Timothy Potter added a comment - I need to create an updated patch out of our internal repo where we originally developed this solution. I can get to this later today.
          Hide
          Erick Erickson added a comment -

          I'm coming late to the party, but wanted to point out this umbrella JIRA: https://issues.apache.org/jira/browse/SOLR-6082.

          The current admin UI carries a bunch of historical design from a very long time ago. It seems to me that SolrCloud admin could be made much more user-friendly if we moved all the SolrCloud stuff to it's own page or something. Having a "select core" dropdown as a menu choice on a page that displays the state of your SolrCloud is...wrong.

          Ideally, I'd like to see all the nodes in my cluster (whether they hosted collections or not). I'd like to ctrl-click on some number of them and be able to create a collection on the selected nodes. I'd like to be able to ctrl-click on a node and add a replica on that node to a collection. I'd like to.... you get the idea. All without having to drop into the shell prompt and use command-line scripts. Or type in a collections API call. We have the infrastructure in place, much of this would be a UI for the Collections API.

          Note, I am not advocating we delay these issues waiting for some grand new design. Mostly I'm wondering if there's enough interest in this kind of thing to start designing a "SolrCloud admin interface". We can use SOLR-6082 as a basis for the discussion if so.

          Show
          Erick Erickson added a comment - I'm coming late to the party, but wanted to point out this umbrella JIRA: https://issues.apache.org/jira/browse/SOLR-6082 . The current admin UI carries a bunch of historical design from a very long time ago. It seems to me that SolrCloud admin could be made much more user-friendly if we moved all the SolrCloud stuff to it's own page or something. Having a "select core" dropdown as a menu choice on a page that displays the state of your SolrCloud is...wrong. Ideally, I'd like to see all the nodes in my cluster (whether they hosted collections or not). I'd like to ctrl-click on some number of them and be able to create a collection on the selected nodes. I'd like to be able to ctrl-click on a node and add a replica on that node to a collection. I'd like to.... you get the idea. All without having to drop into the shell prompt and use command-line scripts. Or type in a collections API call. We have the infrastructure in place, much of this would be a UI for the Collections API. Note, I am not advocating we delay these issues waiting for some grand new design. Mostly I'm wondering if there's enough interest in this kind of thing to start designing a "SolrCloud admin interface". We can use SOLR-6082 as a basis for the discussion if so.
          Hide
          Timothy Potter added a comment -

          Thanks for the pointer to SOLR-6082. Definitely interested in starting to design a SolrCloud admin interface. The work being done in this ticket is mainly for supporting 100's to 1000's of collections from the existing Cloud panel as without some basic nav controls, that panel is unusable when you have many collections. In other words, this will serve as an interim solution for users that have many collections as we work on the design and develop an overhauled SolrCloud Admin UI.

          Show
          Timothy Potter added a comment - Thanks for the pointer to SOLR-6082 . Definitely interested in starting to design a SolrCloud admin interface. The work being done in this ticket is mainly for supporting 100's to 1000's of collections from the existing Cloud panel as without some basic nav controls, that panel is unusable when you have many collections. In other words, this will serve as an interim solution for users that have many collections as we work on the design and develop an overhauled SolrCloud Admin UI.
          Hide
          Mark Miller added a comment -

          So is this only an issue if you have 100+ collections? If so, we should probably update the title / description to be more specific.

          Show
          Mark Miller added a comment - So is this only an issue if you have 100+ collections? If so, we should probably update the title / description to be more specific.
          Hide
          Timothy Potter added a comment -

          No, the status and nav controls work no matter how many collections you have.

          Show
          Timothy Potter added a comment - No, the status and nav controls work no matter how many collections you have.
          Hide
          Mark Miller added a comment -

          #1 is a must for 5473 .

          Right, I'm also only concerned about #1 for SOLR-5473, though I've got nothing against taking it all in if it's ready.

          Show
          Mark Miller added a comment - #1 is a must for 5473 . Right, I'm also only concerned about #1 for SOLR-5473 , though I've got nothing against taking it all in if it's ready.
          Hide
          Timothy Potter added a comment -

          Here's an updated patch that works with the latest patch for SOLR-5473 applied to trunk.

          One thing that's easy to change is that I didn't activate the nav controls if you have less than 10 collections total. If we want the nav controls active all times, that's fine too, I just didn't want to introduce more complexity to the existing UI if it's not needed and I don't think you need paging or filtering if you only have 10 collections (one liner at ZookeeperInfoServlet line 257).

          Show
          Timothy Potter added a comment - Here's an updated patch that works with the latest patch for SOLR-5473 applied to trunk. One thing that's easy to change is that I didn't activate the nav controls if you have less than 10 collections total. If we want the nav controls active all times, that's fine too, I just didn't want to introduce more complexity to the existing UI if it's not needed and I don't think you need paging or filtering if you only have 10 collections (one liner at ZookeeperInfoServlet line 257).
          Hide
          Timothy Potter added a comment -

          Also, I had to clear cache to get the changes to the UI files to update (cloud.js / .css / .html)

          Show
          Timothy Potter added a comment - Also, I had to clear cache to get the changes to the UI files to update (cloud.js / .css / .html)
          Hide
          Erick Erickson added a comment -

          Hmmm, if I apply the patches for SOLR-5473 and this one to trunk, the second patch fails in ZookeeperInfoServlet.java. It doesn't matter whether I apply this patch first or the 5473 first. I used the patch for SOLR-5473 from 3-Aug.

          FWIW..

          Show
          Erick Erickson added a comment - Hmmm, if I apply the patches for SOLR-5473 and this one to trunk, the second patch fails in ZookeeperInfoServlet.java. It doesn't matter whether I apply this patch first or the 5473 first. I used the patch for SOLR-5473 from 3-Aug. FWIW..
          Hide
          Timothy Potter added a comment -

          Updated patch that works with the patch I just updated for SOLR-5473.

          Show
          Timothy Potter added a comment - Updated patch that works with the patch I just updated for SOLR-5473 .
          Hide
          Timothy Potter added a comment - - edited

          Also, here is a link to a screen shot to the nav controls: http://www.dropmocks.com/mGlEAA

          (I still can't get the Jira screenshots feature to work for me)

          Admittedly, the nav panel can be improved from an aesthetic perspective but the functionality should be good to go.

          Show
          Timothy Potter added a comment - - edited Also, here is a link to a screen shot to the nav controls: http://www.dropmocks.com/mGlEAA (I still can't get the Jira screenshots feature to work for me) Admittedly, the nav panel can be improved from an aesthetic perspective but the functionality should be good to go.
          Hide
          ASF subversion and git services added a comment -

          Commit 1624556 from Noble Paul in branch 'dev/trunk'
          [ https://svn.apache.org/r1624556 ]

          split clusterstate.json SOLR-5473, SOLR-5474, SOLR-5810

          Show
          ASF subversion and git services added a comment - Commit 1624556 from Noble Paul in branch 'dev/trunk' [ https://svn.apache.org/r1624556 ] split clusterstate.json SOLR-5473 , SOLR-5474 , SOLR-5810
          Hide
          ASF subversion and git services added a comment -

          Commit 1624650 from hossman@apache.org in branch 'dev/trunk'
          [ https://svn.apache.org/r1624650 ]

          SOLR-5473, SOLR-5474, SOLR-5810 ... NO SVN KEYWORDS! ! !

          Show
          ASF subversion and git services added a comment - Commit 1624650 from hossman@apache.org in branch 'dev/trunk' [ https://svn.apache.org/r1624650 ] SOLR-5473 , SOLR-5474 , SOLR-5810 ... NO SVN KEYWORDS! ! !
          Hide
          Timothy Potter added a comment -

          What was committed is not correct, need to apply the latest patch I posted. Will get it fixed up by tomorrow.

          Show
          Timothy Potter added a comment - What was committed is not correct, need to apply the latest patch I posted. Will get it fixed up by tomorrow.
          Hide
          ASF subversion and git services added a comment -

          Commit 1648621 from Timothy Potter in branch 'dev/trunk'
          [ https://svn.apache.org/r1648621 ]

          SOLR-5810: basic paged nav support for cloud graph panel for many collections.

          Show
          ASF subversion and git services added a comment - Commit 1648621 from Timothy Potter in branch 'dev/trunk' [ https://svn.apache.org/r1648621 ] SOLR-5810 : basic paged nav support for cloud graph panel for many collections.
          Hide
          Timothy Potter added a comment -

          here's the updated patch (committed to trunk) which provides basic paged navigation controls for the cloud graph panel to support many collections. update includes re-establishing the watcher after a Zk session expiration using the OnReconnect interface

          Show
          Timothy Potter added a comment - here's the updated patch (committed to trunk) which provides basic paged navigation controls for the cloud graph panel to support many collections. update includes re-establishing the watcher after a Zk session expiration using the OnReconnect interface
          Hide
          ASF subversion and git services added a comment -

          Commit 1648630 from Timothy Potter in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1648630 ]

          SOLR-5810: backport to branch 5x

          Show
          ASF subversion and git services added a comment - Commit 1648630 from Timothy Potter in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1648630 ] SOLR-5810 : backport to branch 5x
          Hide
          Anshum Gupta added a comment -

          I just started Solr using:

          bin/solr start -e cloud

          and the admin UI doesn't reflect any collection. I can see the core in the dropdown but not in the admin UI.

          New collections are now external by default, so the new collection doesn't show up but then creating a new collection (with explicit stateFormat=2) brings up both the collections on the admin UI.

          Show
          Anshum Gupta added a comment - I just started Solr using: bin/solr start -e cloud and the admin UI doesn't reflect any collection. I can see the core in the dropdown but not in the admin UI. New collections are now external by default, so the new collection doesn't show up but then creating a new collection (with explicit stateFormat=2) brings up both the collections on the admin UI.
          Hide
          Timothy Potter added a comment -

          I don't think this is UI code related ... the /clusterstate.json after running the solr -e cloud command doesn't look correct to me:

          {"gettingstarted":{
              "replicationFactor":"2",
              "router":{"name":"compositeId"},
              "maxShardsPerNode":"2",
              "autoAddReplicas":"false",
              "shards":{
                "shard1":{
                  "range":"80000000-ffffffff",
                  "state":"active",
                  "replicas":{}},
                "shard2":{
                  "range":"0-7fffffff",
                  "state":"active",
                  "replicas":{}}}}}
          

          here's the crazy thing ... if you run bin/solr create -c foo after starting up the example, then /clusterstate.json gets emptied {{ {} }} and the UI renders both collections correctly. So seems like some weird initial state related to stateFormat=2

          Show
          Timothy Potter added a comment - I don't think this is UI code related ... the /clusterstate.json after running the solr -e cloud command doesn't look correct to me: { "gettingstarted" :{ "replicationFactor" : "2" , "router" :{ "name" : "compositeId" }, "maxShardsPerNode" : "2" , "autoAddReplicas" : " false " , "shards" :{ "shard1" :{ "range" : "80000000-ffffffff" , "state" : "active" , "replicas" :{}}, "shard2" :{ "range" : "0-7fffffff" , "state" : "active" , "replicas" :{}}}}} here's the crazy thing ... if you run bin/solr create -c foo after starting up the example, then /clusterstate.json gets emptied {{ {} }} and the UI renders both collections correctly. So seems like some weird initial state related to stateFormat=2
          Hide
          Timothy Potter added a comment -

          This is not a UI issue, but a problem with stateFormat=2 in general, so I opened SOLR-7039 and this one can remain closed / fixed.

          Show
          Timothy Potter added a comment - This is not a UI issue, but a problem with stateFormat=2 in general, so I opened SOLR-7039 and this one can remain closed / fixed.
          Hide
          Anshum Gupta added a comment -

          Bulk close after 5.0 release.

          Show
          Anshum Gupta added a comment - Bulk close after 5.0 release.

            People

            • Assignee:
              Timothy Potter
              Reporter:
              Timothy Potter
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development