Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-7955

Auto create .system collection on first request if it does not exist

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 6.5, 7.0
    • Component/s: None
    • Labels:
      None

      Description

      Why should a user need to create the .system collection manually? It would simplify instructions related to BLOB store if user could assume it is always there.

      1. SOLR-7955.patch
        5 kB
        Noble Paul
      2. SOLR-7955.patch
        0.7 kB
        Jan Høydahl

        Activity

        Hide
        janhoy Jan Høydahl added a comment -

        Reference guide https://cwiki.apache.org/confluence/display/solr/Blob+Store+API requires you to create the collection manually. If instead the Overseer created it on startup with shards=1, replicationfactor=2, then the refguide could instead document that if you require more shards or replicas then you should modify or recreate the collection manually.

        Show
        janhoy Jan Høydahl added a comment - Reference guide https://cwiki.apache.org/confluence/display/solr/Blob+Store+API requires you to create the collection manually. If instead the Overseer created it on startup with shards=1, replicationfactor=2, then the refguide could instead document that if you require more shards or replicas then you should modify or recreate the collection manually.
        Hide
        noble.paul Noble Paul added a comment -

        Yes, this was the original plan. I was waiting for the feature to settle

        • In a cluster when a create collection is invoked check if .system collection exists. If not create a .system collection with shards=1, and replicationFactor=2
        • if the first collection to be created is .system then it's fine
        Show
        noble.paul Noble Paul added a comment - Yes, this was the original plan. I was waiting for the feature to settle In a cluster when a create collection is invoked check if .system collection exists. If not create a .system collection with shards=1, and replicationFactor=2 if the first collection to be created is .system then it's fine
        Hide
        janhoy Jan Høydahl added a comment -

        What if admin chooses to upload a bunch of jars to blob store right after install, and then someone else, perhaps an application, will later create a bunch of collections. Then the .system collection must exist independent of collection CREATE requests.

        Likewise, it would be nice if a collection could be created with a flag "bind all gobal runtime libs", much like how the current static $SOLR_HOME/lib folder works, i.e. all collections will have access to all the global libs. E.g. if a collections was created with &property.runtimeLibs=all then you would not, as a collection creator, need to know the keys by which they are registered and not need to - you'd get them all (latest versions) on classpath. For many smaller setups this would simplify things.

        Show
        janhoy Jan Høydahl added a comment - What if admin chooses to upload a bunch of jars to blob store right after install, and then someone else, perhaps an application, will later create a bunch of collections. Then the .system collection must exist independent of collection CREATE requests. Likewise, it would be nice if a collection could be created with a flag "bind all gobal runtime libs", much like how the current static $SOLR_HOME/lib folder works, i.e. all collections will have access to all the global libs. E.g. if a collections was created with &property.runtimeLibs=all then you would not, as a collection creator, need to know the keys by which they are registered and not need to - you'd get them all (latest versions) on classpath. For many smaller setups this would simplify things.
        Hide
        noble.paul Noble Paul added a comment -

        What if admin chooses to upload a bunch of jars to blob store right after install, and then someone else, perhaps an application, will later create a bunch of collections. Then the .system collection must exist independent of collection CREATE requests.

        If admin wants to upload a bunch of jars before collection CREATE , go ahead and create the .system collection first and upload all jars

        Likewise, it would be nice if a collection could be created with a flag "bind all gobal runtime libs",

        I don't think you understand the current feature fully.

        There is nothing called all "runtime libs" . blob store has more things than just jars. And there are many versions of each jars. User MUST specify which version of the jar he wants to use.
        The idea of "latest" is NOT possible , it can mean that the same collection in two different nodes may run different versions of a library.

        Show
        noble.paul Noble Paul added a comment - What if admin chooses to upload a bunch of jars to blob store right after install, and then someone else, perhaps an application, will later create a bunch of collections. Then the .system collection must exist independent of collection CREATE requests. If admin wants to upload a bunch of jars before collection CREATE , go ahead and create the .system collection first and upload all jars Likewise, it would be nice if a collection could be created with a flag "bind all gobal runtime libs", I don't think you understand the current feature fully. There is nothing called all "runtime libs" . blob store has more things than just jars. And there are many versions of each jars. User MUST specify which version of the jar he wants to use. The idea of "latest" is NOT possible , it can mean that the same collection in two different nodes may run different versions of a library.
        Hide
        janhoy Jan Høydahl added a comment -

        Ok, understand that blob store is just a generic key/value store and knows nothing about the data. So any smartness here would need to be added on a layer above. Let's defer that to another jira.

        Show
        janhoy Jan Høydahl added a comment - Ok, understand that blob store is just a generic key/value store and knows nothing about the data. So any smartness here would need to be added on a layer above. Let's defer that to another jira.
        Hide
        elyograg Shawn Heisey added a comment - - edited

        Devil's advocate thoughts:

        If this is auto-created when the first node is started, then it will be a non-redundant collection. If we wait for two nodes, then a single-node test install will not have it. If we automatically add replicas as new nodes are brought up, the user might be very unhappy with that decision.

        I do like the idea of creating this collection automatically, but its behavior must be configurable, with sensible defaults. Any actions taken (or explicitly NOT taken) should probably be logged at WARN so that someone looking in the logging tab of the admin UI can see them.

        Show
        elyograg Shawn Heisey added a comment - - edited Devil's advocate thoughts: If this is auto-created when the first node is started, then it will be a non-redundant collection. If we wait for two nodes, then a single-node test install will not have it. If we automatically add replicas as new nodes are brought up, the user might be very unhappy with that decision. I do like the idea of creating this collection automatically, but its behavior must be configurable, with sensible defaults. Any actions taken (or explicitly NOT taken) should probably be logged at WARN so that someone looking in the logging tab of the admin UI can see them.
        Hide
        markrmiller@gmail.com Mark Miller added a comment -

        Personally, I like the idea that the majority of people not using this feature don't have to deal with this internal collection. If they decide they want to use the blob api, they create the collection - simple.

        Show
        markrmiller@gmail.com Mark Miller added a comment - Personally, I like the idea that the majority of people not using this feature don't have to deal with this internal collection. If they decide they want to use the blob api, they create the collection - simple.
        Hide
        andyetitmoves Ramkumar Aiyengar added a comment -

        I agree, there's a footprint to adding a collection, and people who don't need this feature shouldn't have to bear it. If we want to make the feature easier, may be lazily on first use (if possible) is a better alternative than on first start..

        Show
        andyetitmoves Ramkumar Aiyengar added a comment - I agree, there's a footprint to adding a collection, and people who don't need this feature shouldn't have to bear it. If we want to make the feature easier, may be lazily on first use (if possible) is a better alternative than on first start..
        Hide
        janhoy Jan Høydahl added a comment -

        may be lazily on first use (if possible) is a better alternative than on first start..

        "First use" in this context I guess is first data upload, e.g.

        curl -X POST ... --data-binary @test1.jar http://localhost:8983/solr/.system/blob/test
        

        Is it even possible to intercept a POST to .system if it does not exist, or is the special /blob handler only registered when collection is created?

        Show
        janhoy Jan Høydahl added a comment - may be lazily on first use (if possible) is a better alternative than on first start.. "First use" in this context I guess is first data upload, e.g. curl -X POST ... --data-binary @test1.jar http://localhost:8983/solr/.system/blob/test Is it even possible to intercept a POST to .system if it does not exist, or is the special /blob handler only registered when collection is created?
        Hide
        andyetitmoves Ramkumar Aiyengar added a comment -

        Should certainly be possible. Actually there's the concept of lazy cores already which probably does this already, but I am not sure..

        Show
        andyetitmoves Ramkumar Aiyengar added a comment - Should certainly be possible. Actually there's the concept of lazy cores already which probably does this already, but I am not sure..
        Hide
        noble.paul Noble Paul added a comment -

        Lazy cores is different. We can auto create the collection when the first request comes. The core container has to be aware of this special collection

        Show
        noble.paul Noble Paul added a comment - Lazy cores is different. We can auto create the collection when the first request comes. The core container has to be aware of this special collection
        Hide
        janhoy Jan Høydahl added a comment -

        Changed issue summary to "Auto create .system collection on first request if it does not exist". Agree this is a more elegant solution.

        Show
        janhoy Jan Høydahl added a comment - Changed issue summary to "Auto create .system collection on first request if it does not exist". Agree this is a more elegant solution.
        Hide
        janhoy Jan Høydahl added a comment -

        Simple patch which just logs a line in HttpSolrCall when we have detected .system collection not exists. Suppose we could put some auto-create logic there?

        Show
        janhoy Jan Høydahl added a comment - Simple patch which just logs a line in HttpSolrCall when we have detected .system collection not exists. Suppose we could put some auto-create logic there?
        Hide
        elyograg Shawn Heisey added a comment -

        Another TL;DR comment:

        Something I touched on earlier: What will the default value for replicationFactor be when the collection is automatically built? One idea, but I'm far from saying it's the best idea, is to set the default to the number of servers in the cloud.

        If the replica count is hard-coded to 2, or uses the number of servers, the following scenario would result either in no .system collection at all (failed create due to insufficient servers) or no redundancy: A user sets up a cloud, initially with one server, then creates a collection on that cloud to try things out before adding more servers and setting up their real collection(s).

        I like the idea of not making the user do things manually, but if a user proceeds in the cautious manner I just described, I worry that we'll put that user in a bad situation at a later date when the server containing their .system collection dies. We can create good documentation, but unless the user thinks to LOOK for that documentation, they might get a nasty surprise one day.

        Related: The reference guide probably needs a section describing problems that can arise from a lack of understanding, and how to remedy them. A possible section title: "Gotchas: Solving problems you may not know you have"

        Show
        elyograg Shawn Heisey added a comment - Another TL;DR comment: Something I touched on earlier: What will the default value for replicationFactor be when the collection is automatically built? One idea, but I'm far from saying it's the best idea, is to set the default to the number of servers in the cloud. If the replica count is hard-coded to 2, or uses the number of servers, the following scenario would result either in no .system collection at all (failed create due to insufficient servers) or no redundancy: A user sets up a cloud, initially with one server, then creates a collection on that cloud to try things out before adding more servers and setting up their real collection(s). I like the idea of not making the user do things manually, but if a user proceeds in the cautious manner I just described, I worry that we'll put that user in a bad situation at a later date when the server containing their .system collection dies. We can create good documentation, but unless the user thinks to LOOK for that documentation, they might get a nasty surprise one day. Related: The reference guide probably needs a section describing problems that can arise from a lack of understanding, and how to remedy them. A possible section title: "Gotchas: Solving problems you may not know you have"
        Hide
        janhoy Jan Høydahl added a comment -

        Defaulting to number of live nodes in cluster would probably do. Then we'd need to document that manual ADDREPLICA calls are necessary to achieve redundancy if you start off with one node and later add nodes.

        Ideally we should re-visit the former vision of being able to configure a desired state for a collection, which the cluster (Overseer) will try to achieve. Say you configure replicationFactor=3&stateType=desired in a one-node cluster. Then you add nodes, and the Overseer issues ADDREPLICA commands until condition is reached. Next, we could allow variables/functions instead of absolute numbers, i.e. replicationFactor=nodeCount(). Of course there are many issues to such an approach too, e.g. actions should probably only trigger N seconds after the last add/remove command.

        Show
        janhoy Jan Høydahl added a comment - Defaulting to number of live nodes in cluster would probably do. Then we'd need to document that manual ADDREPLICA calls are necessary to achieve redundancy if you start off with one node and later add nodes. Ideally we should re-visit the former vision of being able to configure a desired state for a collection, which the cluster (Overseer) will try to achieve. Say you configure replicationFactor=3&stateType=desired in a one-node cluster. Then you add nodes, and the Overseer issues ADDREPLICA commands until condition is reached. Next, we could allow variables/functions instead of absolute numbers, i.e. replicationFactor=nodeCount() . Of course there are many issues to such an approach too, e.g. actions should probably only trigger N seconds after the last add/remove command.
        Hide
        noble.paul Noble Paul added a comment -

        Let's limit the scope to creating the .system collection during the first request. I would say let's keep the replicationfactor=Math.min(3,totalNodes) . For the time being, it should be good enough. Users can do ADDREPLICA later

        Show
        noble.paul Noble Paul added a comment - Let's limit the scope to creating the .system collection during the first request. I would say let's keep the replicationfactor=Math.min(3,totalNodes) . For the time being, it should be good enough. Users can do ADDREPLICA later
        Hide
        noble.paul Noble Paul added a comment -

        I'm planning to commit this as soon as I add a test

        Show
        noble.paul Noble Paul added a comment - I'm planning to commit this as soon as I add a test
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit e200b8a2a418cdb145acb51d1181b1b60362a926 in lucene-solr's branch refs/heads/master from Noble Paul
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=e200b8a ]

        SOLR-7955: Auto create .system collection on first request if it does not exist

        Show
        jira-bot ASF subversion and git services added a comment - Commit e200b8a2a418cdb145acb51d1181b1b60362a926 in lucene-solr's branch refs/heads/master from Noble Paul [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=e200b8a ] SOLR-7955 : Auto create .system collection on first request if it does not exist
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 3e44928f490c34666107e9bd6393020be160865f in lucene-solr's branch refs/heads/master from Noble Paul
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=3e44928 ]

        SOLR-7955: making it easy to commit to branch_6x

        Show
        jira-bot ASF subversion and git services added a comment - Commit 3e44928f490c34666107e9bd6393020be160865f in lucene-solr's branch refs/heads/master from Noble Paul [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=3e44928 ] SOLR-7955 : making it easy to commit to branch_6x
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit ff1a9e923530cd044eebcee5f7dcca77e26b7d0c in lucene-solr's branch refs/heads/master from Noble Paul
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=ff1a9e9 ]

        SOLR-7955: further optimization to avoid zk lookup

        Show
        jira-bot ASF subversion and git services added a comment - Commit ff1a9e923530cd044eebcee5f7dcca77e26b7d0c in lucene-solr's branch refs/heads/master from Noble Paul [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=ff1a9e9 ] SOLR-7955 : further optimization to avoid zk lookup
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit ec778c5c8f6b3bcb551a3ae9999c73f664e2de90 in lucene-solr's branch refs/heads/branch_6x from Noble Paul
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=ec778c5 ]

        SOLR-7955: Auto create .system collection on first request if it does not exist

        Show
        jira-bot ASF subversion and git services added a comment - Commit ec778c5c8f6b3bcb551a3ae9999c73f664e2de90 in lucene-solr's branch refs/heads/branch_6x from Noble Paul [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=ec778c5 ] SOLR-7955 : Auto create .system collection on first request if it does not exist
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit afae0d971867216d1f02a96395ba939a37c8d86e in lucene-solr's branch refs/heads/branch_6x from Noble Paul
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=afae0d9 ]

        SOLR-7955: Auto create .system collection on first request if it does not exist

        Show
        jira-bot ASF subversion and git services added a comment - Commit afae0d971867216d1f02a96395ba939a37c8d86e in lucene-solr's branch refs/heads/branch_6x from Noble Paul [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=afae0d9 ] SOLR-7955 : Auto create .system collection on first request if it does not exist

          People

          • Assignee:
            noble.paul Noble Paul
            Reporter:
            janhoy Jan Høydahl
          • Votes:
            1 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development