Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.0-ALPHA
    • Component/s: web gui
    • Labels:
      None
    1. SOLR-3174-rgraph.png
      96 kB
      Stefan Matheis (steffkes)
    2. SOLR-3174-graph.png
      109 kB
      Stefan Matheis (steffkes)
    3. SOLR-3174.patch
      16 kB
      Stefan Matheis (steffkes)
    4. SOLR-3174.patch
      16 kB
      Stefan Matheis (steffkes)
    5. SOLR-3174.patch
      17 kB
      Stefan Matheis (steffkes)
    6. SOLR-3174-rgraph.png
      140 kB
      Stefan Matheis (steffkes)
    7. SOLR-3174-graph.png
      148 kB
      Stefan Matheis (steffkes)
    8. SOLR-3174.patch
      13 kB
      Stefan Matheis (steffkes)
    9. SOLR-3174.patch
      11 kB
      Stefan Matheis (steffkes)
    10. SOLR-3174-rgraph.png
      229 kB
      Stefan Matheis (steffkes)
    11. SOLR-3174-graph.png
      199 kB
      Stefan Matheis (steffkes)

      Issue Links

        Activity

        Hide
        Ryan McKinley added a comment -

        There are a few libraries that could make this relativly straightforward and good looking:

        http://flare.prefuse.org/demo
        http://jsplumb.org/jquery/chartDemo.html
        http://neyric.github.com/wireit/

        Show
        Ryan McKinley added a comment - There are a few libraries that could make this relativly straightforward and good looking: http://flare.prefuse.org/demo http://jsplumb.org/jquery/chartDemo.html http://neyric.github.com/wireit/
        Hide
        Tommaso Teofili added a comment -

        yes this'd be a very nice improvement

        Show
        Tommaso Teofili added a comment - yes this'd be a very nice improvement
        Show
        Stefan Matheis (steffkes) added a comment - Just to bring two more libraries: http://arborjs.org/halfviz/ http://thejit.org/static/v20/Jit/Examples/Spacetree/example1.html
        Hide
        Mark Miller added a comment -

        +1 - this would be super helpful

        Show
        Mark Miller added a comment - +1 - this would be super helpful
        Hide
        Stefan Matheis (steffkes) added a comment -

        I'll try to launch a small Cloud on my local VMWare and build an example w/ each of these libraries .. so we'll see which fits our requirements best - will need your input on this, for sure ;>

        Show
        Stefan Matheis (steffkes) added a comment - I'll try to launch a small Cloud on my local VMWare and build an example w/ each of these libraries .. so we'll see which fits our requirements best - will need your input on this, for sure ;>
        Hide
        Mark Miller added a comment -

        If you are on a unix machine, then in /solr/cloud-dev you could just run solrcloud-start.sh and it starts up a 2 shard, 4 node cluster automatically. Unfortunately, no windows bat files currently

        Show
        Mark Miller added a comment - If you are on a unix machine, then in /solr/cloud-dev you could just run solrcloud-start.sh and it starts up a 2 shard, 4 node cluster automatically. Unfortunately, no windows bat files currently
        Hide
        Stefan Matheis (steffkes) added a comment - - edited

        The first ones, based on this clusterstate.json:

        these are really just examples, no real styling. i guess it's more a question about general layout? freaky particular-systems like arbor.js or more classical tree-variant like jit? js-plump has a nice rendering, but the styling is completely manually .. the first ones compute the positions of each shard/node item automatically.

        Show
        Stefan Matheis (steffkes) added a comment - - edited The first ones, based on this clusterstate.json : http://files.mathe.is/solr-admin/cloud-state/arbor/ http://files.mathe.is/solr-admin/cloud-state/jit/ http://files.mathe.is/solr-admin/cloud-state/js-plumb/ these are really just examples, no real styling. i guess it's more a question about general layout? freaky particular-systems like arbor.js or more classical tree-variant like jit? js-plump has a nice rendering, but the styling is completely manually .. the first ones compute the positions of each shard/node item automatically.
        Hide
        Ryan McKinley added a comment -

        I like the JIT layout best. Would be cool to be able to switch between the tree view:
        http://files.mathe.is/solr-admin/cloud-state/jit/

        and a radial graph view:
        http://thejit.org/static/v20/Jit/Examples/RGraph/example1.html

        Show
        Ryan McKinley added a comment - I like the JIT layout best. Would be cool to be able to switch between the tree view: http://files.mathe.is/solr-admin/cloud-state/jit/ and a radial graph view: http://thejit.org/static/v20/Jit/Examples/RGraph/example1.html
        Hide
        Mark Miller added a comment -

        Cool - they all look great! I kind of like the arbor view myself.

        One thing to keep in mind is that it might be cool to have clickable linkes involved if that affects the choice - eg if you could click on some icon or text that is part of the node and go to the admin for that node, that would be really useful.

        Show
        Mark Miller added a comment - Cool - they all look great! I kind of like the arbor view myself. One thing to keep in mind is that it might be cool to have clickable linkes involved if that affects the choice - eg if you could click on some icon or text that is part of the node and go to the admin for that node, that would be really useful.
        Hide
        Stefan Matheis (steffkes) added a comment -

        Created another one for ryan's radical view: http://files.mathe.is/solr-admin/cloud-state/jit-rgraph/ - i've added another shard w/ three nodes, just for layout purposes .. otherwise the generated layout looks a bit odd ;o

        To continue w/ the integration of that view, it would be very helpful to get some "real life data" .. would that be possible? what would a cluster normally look like? how many shards w/ how many nodes? so i'd get a feeling on how large the Graphs could be .. where they could be integrated in the admin-ui and so on

        Show
        Stefan Matheis (steffkes) added a comment - Created another one for ryan's radical view: http://files.mathe.is/solr-admin/cloud-state/jit-rgraph/ - i've added another shard w/ three nodes, just for layout purposes .. otherwise the generated layout looks a bit odd ;o To continue w/ the integration of that view, it would be very helpful to get some "real life data" .. would that be possible? what would a cluster normally look like? how many shards w/ how many nodes? so i'd get a feeling on how large the Graphs could be .. where they could be integrated in the admin-ui and so on
        Hide
        Erick Erickson added a comment -

        Way cool! I suspect that we should give them all the space they can have, but plan for 5 shards, three replicas? That covers quite a large installation actually. Almost immediately someone will have 20 shards and 10 replicas each, but at some point it gets ridiculous.

        How about opening up a new tab and giving them the whole screen?

        As you can tell, there's a dearth of "real life" data here, all we can do is guess.

        Show
        Erick Erickson added a comment - Way cool! I suspect that we should give them all the space they can have, but plan for 5 shards, three replicas? That covers quite a large installation actually. Almost immediately someone will have 20 shards and 10 replicas each, but at some point it gets ridiculous. How about opening up a new tab and giving them the whole screen? As you can tell, there's a dearth of "real life" data here, all we can do is guess.
        Hide
        Mark Miller added a comment -

        I would guestimate common distributed setups might have 3-15 shards (favoring lower). Common replication factor might be 2 or 3.

        Of course those with billions of docs collections might get into 100+ shards area - again replication factor is still going to probably be 3-5 max I would guess.

        Show
        Mark Miller added a comment - I would guestimate common distributed setups might have 3-15 shards (favoring lower). Common replication factor might be 2 or 3. Of course those with billions of docs collections might get into 100+ shards area - again replication factor is still going to probably be 3-5 max I would guess.
        Hide
        Cody Young added a comment -

        18 shards, 3-5 replicas here.

        Show
        Cody Young added a comment - 18 shards, 3-5 replicas here.
        Hide
        Stefan Matheis (steffkes) added a comment -

        Discovered d3.js, which as also a Tree/Graph Sample: http://mbostock.github.com/d3/ex/tree.html .. will do another Sample for that.

        Show
        Stefan Matheis (steffkes) added a comment - Discovered d3.js, which as also a Tree/Graph Sample: http://mbostock.github.com/d3/ex/tree.html .. will do another Sample for that.
        Hide
        Stefan Matheis (steffkes) added a comment -

        While working on the Integration, i tried various States for the Cloud .. so for example, after running the Sample-Cluster, i continued working with only one (Master-Node).

        That Content of clusterstate.json still remains the same, i guess because i have not removed the files in solr/zoo_data, right? If this will be also possible in Production Use-Cases .. is there a need to verify the list of given Servers against the live_nodes Folders in zk? Or are we fine just relying on clusterstate.json's Infos and displaying them in Tree/Graph View?

        Show
        Stefan Matheis (steffkes) added a comment - While working on the Integration, i tried various States for the Cloud .. so for example, after running the Sample-Cluster, i continued working with only one (Master-Node). That Content of clusterstate.json still remains the same, i guess because i have not removed the files in solr/zoo_data , right? If this will be also possible in Production Use-Cases .. is there a need to verify the list of given Servers against the live_nodes Folders in zk? Or are we fine just relying on clusterstate.json's Infos and displaying them in Tree/Graph View?
        Hide
        Mark Miller added a comment -

        i guess because i have not removed the files in solr/zoo_data, right?

        Yeah, exactly.

        is there a need to verify the list of given Servers against the live_nodes Folders in zk? Or are we fine just relying on clusterstate.json's Infos and displaying them in Tree/Graph View?

        Nope, I'd just stick to the clusterstate.json - that is the truth - the truth is just a little whacky if you try and start over without clearing zoo_data.

        Show
        Mark Miller added a comment - i guess because i have not removed the files in solr/zoo_data, right? Yeah, exactly. is there a need to verify the list of given Servers against the live_nodes Folders in zk? Or are we fine just relying on clusterstate.json's Infos and displaying them in Tree/Graph View? Nope, I'd just stick to the clusterstate.json - that is the truth - the truth is just a little whacky if you try and start over without clearing zoo_data.
        Hide
        Mark Miller added a comment -

        Discovered d3.js

        Nice - that one looks pretty cool too.

        Show
        Mark Miller added a comment - Discovered d3.js Nice - that one looks pretty cool too.
        Hide
        Stefan Matheis (steffkes) added a comment -

        So, d3-samples finally arrived ;o

        Tree: small, large
        Radial: small, large

        I'll go with this for the first time .. and we'll see how good it fits our needs

        Show
        Stefan Matheis (steffkes) added a comment - So, d3-samples finally arrived ;o Tree: small , large Radial: small , large I'll go with this for the first time .. and we'll see how good it fits our needs
        Hide
        Stefan Matheis (steffkes) added a comment -

        Attached the current Patch and two Screenshots. Will try to finish the Radial-View over the Weekend. But anyway, is that how you'd like it?

        Show
        Stefan Matheis (steffkes) added a comment - Attached the current Patch and two Screenshots. Will try to finish the Radial-View over the Weekend. But anyway, is that how you'd like it?
        Hide
        Stefan Matheis (steffkes) added a comment -

        Updated the Patch and the Screenshots. Radial-View is now working as expected. Also improved the displayed Hostname, if all have the same protocol, it's skipped - same for ports and directories.

        Show
        Stefan Matheis (steffkes) added a comment - Updated the Patch and the Screenshots. Radial-View is now working as expected. Also improved the displayed Hostname, if all have the same protocol, it's skipped - same for ports and directories.
        Hide
        Mark Miller added a comment -

        This is Awesome Stefan - thanks a million!

        It would be cool to add some additional features (like being able to click a node and go to its solr admin page) - but that is just gravy for further issues probably - this is fantastic stuff - lets put it in!

        Show
        Mark Miller added a comment - This is Awesome Stefan - thanks a million! It would be cool to add some additional features (like being able to click a node and go to its solr admin page) - but that is just gravy for further issues probably - this is fantastic stuff - lets put it in!
        Hide
        Mark Miller added a comment -

        (like being able to click a node and go to its solr admin page)

        I was only looking at the screenshots - just applied the patch and tried it out and I see you already did that!

        Show
        Mark Miller added a comment - (like being able to click a node and go to its solr admin page) I was only looking at the screenshots - just applied the patch and tried it out and I see you already did that!
        Hide
        Mark Miller added a comment -

        What are you currently doing for status indication?

        That is actually slightly complicated because its determined by a mix of whether or not the nodes is under /live_nodes and what its status property is...

        Show
        Mark Miller added a comment - What are you currently doing for status indication? That is actually slightly complicated because its determined by a mix of whether or not the nodes is under /live_nodes and what its status property is...
        Hide
        Stefan Matheis (steffkes) added a comment -

        What are you currently doing for status indication?

        Aaaww, i knew that ;p Go ahead and tell me, how it should work. At the moment it's indeed using plain state-property from clusterstate.json.

        Show
        Stefan Matheis (steffkes) added a comment - What are you currently doing for status indication? Aaaww, i knew that ;p Go ahead and tell me, how it should work. At the moment it's indeed using plain state -property from clusterstate.json .
        Hide
        Mark Miller added a comment -

        The reason it's kind of complicated is because when a node is shutdown or dies, it's published state does not get updated...so a down node could be listed as active. Because of this, the true state is a mix of /live_nodes and the state. If the node_name is listed under /live_nodes, then the state is the truth - if the node is not listed under /live_nodes, the state for the node could be anything and indicates nothing - the node is down and not part of the cluster.

        For example:

        [{
            "num_shards":"2",
            "shard":"shard1",
            "state":"active",
            "core":"",
            "collection":"collection1",
            "node_name":"halfmetal:7574_solr",
            "base_url":"http://halfmetal:7574/solr"}]
        

        This node is listed as active, but looking at /live_nodes:

        /live_nodes/halfmetal:8983_solr
        /live_nodes/halfmetal:9983_solr
        /live_nodes/halfmetal:3983_solr
        

        It's 'node_name' is not listed, and so the node should be considered 'gone'. This is actually different than the 'down' state - a node might be listed in /live_nodes, meaning it's part of the cluster, but have a state of 'down' because it could not properly start.

        Show
        Mark Miller added a comment - The reason it's kind of complicated is because when a node is shutdown or dies, it's published state does not get updated...so a down node could be listed as active. Because of this, the true state is a mix of /live_nodes and the state. If the node_name is listed under /live_nodes, then the state is the truth - if the node is not listed under /live_nodes, the state for the node could be anything and indicates nothing - the node is down and not part of the cluster. For example: [{ "num_shards":"2", "shard":"shard1", "state":"active", "core":"", "collection":"collection1", "node_name":"halfmetal:7574_solr", "base_url":"http://halfmetal:7574/solr"}] This node is listed as active, but looking at /live_nodes: /live_nodes/halfmetal:8983_solr /live_nodes/halfmetal:9983_solr /live_nodes/halfmetal:3983_solr It's 'node_name' is not listed, and so the node should be considered 'gone'. This is actually different than the 'down' state - a node might be listed in /live_nodes, meaning it's part of the cluster, but have a state of 'down' because it could not properly start.
        Hide
        Stefan Matheis (steffkes) added a comment -

        Updated Patch, the implemented Logic looks like this:

        if( live_nodes[node_name] && 'active' === state ) { status = 'active'; }
        else if( live_nodes[node_name] )                  { status = 'down'; }
        else                                              { status = 'gone'; }

        As far as i understand .. that is specific enough to get the right state shown, right?

        Perhaps we need to tweak the colors in the UI to make it more clear if something is active/gone/down and which one is the master? let me know what you think

        Show
        Stefan Matheis (steffkes) added a comment - Updated Patch, the implemented Logic looks like this: if ( live_nodes[node_name] && 'active' === state ) { status = 'active'; } else if ( live_nodes[node_name] ) { status = 'down'; } else { status = 'gone'; } As far as i understand .. that is specific enough to get the right state shown, right? Perhaps we need to tweak the colors in the UI to make it more clear if something is active/gone/down and which one is the master? let me know what you think
        Hide
        Mark Miller added a comment -

        Perhaps we need to tweak the colors in the UI to make it more clear if something is active/gone/down and which one is the master?

        +1

        I think that logic looks okay. Another possible status is Recovering.

        An idea for colors:
        active - green
        down - red
        gone - gray
        recovering - yellow

        Show
        Mark Miller added a comment - Perhaps we need to tweak the colors in the UI to make it more clear if something is active/gone/down and which one is the master? +1 I think that logic looks okay. Another possible status is Recovering. An idea for colors: active - green down - red gone - gray recovering - yellow
        Hide
        Mark Miller added a comment -

        just looked - currently there is also recovery_failed.

        So perhaps down should be orange (it's not sure to be terrible if you are down - you might still recover)
        and then recovery_failed should be red.

        Show
        Mark Miller added a comment - just looked - currently there is also recovery_failed. So perhaps down should be orange (it's not sure to be terrible if you are down - you might still recover) and then recovery_failed should be red.
        Hide
        Erick Erickson added a comment -

        Hmmm, this doesn't apply for me against a new trunk, cloud.js has a bunch of "hunk failed" problems. Which is puzzling since that file hasn't been updated in SVN since before the last version of the patch.

        Stefan:

        If it's easy, could you make a new patch? My need isn't that urgent, I just wanted to take a look so if it's anything other than doing an update and making a new patch, don't bother on my account....

        Thanks
        Erick

        Show
        Erick Erickson added a comment - Hmmm, this doesn't apply for me against a new trunk, cloud.js has a bunch of "hunk failed" problems. Which is puzzling since that file hasn't been updated in SVN since before the last version of the patch. Stefan: If it's easy, could you make a new patch? My need isn't that urgent, I just wanted to take a look so if it's anything other than doing an update and making a new patch, don't bother on my account.... Thanks Erick
        Hide
        Stefan Matheis (steffkes) added a comment -

        Mark, this should be possible - can we decide about the priority of each state? Just for starting the discussion, what i've understood right now:

        active          : green
        recovering      : yellow
        down            : orange
        recovery_failed : red
        gone            : gray

        And the other question: how to check the correct state? Which one includes a check against live_nodes and which not?

        Show
        Stefan Matheis (steffkes) added a comment - Mark, this should be possible - can we decide about the priority of each state? Just for starting the discussion, what i've understood right now: active : green recovering : yellow down : orange recovery_failed : red gone : gray And the other question: how to check the correct state? Which one includes a check against live_nodes and which not?
        Hide
        Stefan Matheis (steffkes) added a comment -

        This one is for you Erick ;>

        Show
        Stefan Matheis (steffkes) added a comment - This one is for you Erick ;>
        Hide
        Erick Erickson added a comment -

        Cool, applied cleanly.....

        Thanks

        Show
        Erick Erickson added a comment - Cool, applied cleanly..... Thanks
        Hide
        Mark Miller added a comment - - edited

        if not in live_nodes:
        it could have any state - ignore the state and make the color gray

        if in live_nodes:
        Use the following color based on the state string in the first column.

        active          : green
        recovering      : yellow
        down            : orange
        recovery_failed : red
        
        Show
        Mark Miller added a comment - - edited if not in live_nodes: it could have any state - ignore the state and make the color gray if in live_nodes: Use the following color based on the state string in the first column. active : green recovering : yellow down : orange recovery_failed : red
        Hide
        Stefan Matheis (steffkes) added a comment -

        So, there we go, updated patch with the logic, previously described

        Show
        Stefan Matheis (steffkes) added a comment - So, there we go, updated patch with the logic, previously described
        Hide
        Stefan Matheis (steffkes) added a comment -

        Updated Screens, to show what it will typically look like.

        Show
        Stefan Matheis (steffkes) added a comment - Updated Screens, to show what it will typically look like.
        Hide
        Stefan Matheis (steffkes) added a comment -

        I'll commit this during the day - we can discuss further enhancements in another ticket, k? :>

        Show
        Stefan Matheis (steffkes) added a comment - I'll commit this during the day - we can discuss further enhancements in another ticket, k? :>
        Hide
        Stefan Matheis (steffkes) added a comment -

        Committed current state in r1328330. Let me know, what you think about it

        Show
        Stefan Matheis (steffkes) added a comment - Committed current state in r1328330. Let me know, what you think about it
        Hide
        Mark Miller added a comment -

        This is fantastic Stefan - I can't thank you enough for the work you have put in to the new admin UI. It is light years ahead of what we had.

        Just popped back to this issue because I've been doing some testing, and it looks like even though I have nodes that are trying to recover, everything looks green and happy. I'll try and do some more debugging when I get a moment.

        Show
        Mark Miller added a comment - This is fantastic Stefan - I can't thank you enough for the work you have put in to the new admin UI. It is light years ahead of what we had. Just popped back to this issue because I've been doing some testing, and it looks like even though I have nodes that are trying to recover, everything looks green and happy. I'll try and do some more debugging when I get a moment.
        Hide
        Mark Miller added a comment -

        Whoops - I think I just missed the real issue - I'm working with 2 collections (collection1 and collection2), but only one of them (collection1) is showing up. collection2 has the recoveries occurring. I'll file a new JIRA.

        Show
        Mark Miller added a comment - Whoops - I think I just missed the real issue - I'm working with 2 collections (collection1 and collection2), but only one of them (collection1) is showing up. collection2 has the recoveries occurring. I'll file a new JIRA.
        Hide
        Mark Miller added a comment -

        Another thing we should probably do is add a key for the meaning of the colors.

        Show
        Mark Miller added a comment - Another thing we should probably do is add a key for the meaning of the colors.
        Hide
        Stefan Matheis (steffkes) added a comment -

        Another thing we should probably do is add a key for the meaning of the colors.

        oO didn't see this comment yet ... but now we have one, coming with SOLR-3915 =)

        Show
        Stefan Matheis (steffkes) added a comment - Another thing we should probably do is add a key for the meaning of the colors. oO didn't see this comment yet ... but now we have one, coming with SOLR-3915 =)

          People

          • Assignee:
            Stefan Matheis (steffkes)
            Reporter:
            Ryan McKinley
          • Votes:
            3 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development