Details

      Description

      Today it is like this

      • SolrServer.request returns NamedList<Object>
      • SolrRequest.process returns SolrResponse
      • Public methods on SolrServer like addX, optimize, commit, queryX etc. returns subclasses of SolrResponse (e.g. "add" returns UpdateResponse)
      • etc

      This is all synchronous - that is, the calling thread of those methods will wait for the response before being able to continue. I believe the industry today agrees that "operations" like client-server network-requireing operations should be done asynchronously seens from the client API. Therefore basically we should change those methods

      • SolrServer.request returns Future<NamedList<Object>>
      • SolrRequest.process returns Future<SolrResponse>
      • SolrServer.xxx returns Future<subclass of SolrResponse>
        and make the appropriate changes in the implementations below.

      My main argument for this right now, is that ConcurrentUpdateSolrServer really is not able to hand over responses to the calling client. Guess that it is also the reason why it is only a "Update"SolrServer and not a complete SolrServer (being able to do queries etc.) - updates does not require sending responses (except primitive errors) back to the client, queries etc does. Now that we do "finegrained error propagation" (SOLR-3382) in order to send "unique key constraint" and "versioning"-errors (SOLR-3173 and SOLR-3178) back to the client in responses to update-request, suddenly it is not true anymore that updates does not require sending responses back to the client.
      Making the changes suggested above (returning Futures) would

      • Allow ConcurrentUpdateSolrServer to be used for updates potentially resulting in "unique key constraint"- and "versioning"-errors
      • Allow ConcurrentUpdateSolrServer to become ConcurrentSolrServer - also being able to do queries etc
      • Do cool stuff like SOLR-3384
      • Make SolrJ more modern with respect to asynchronous principles
      1. SOLR-3383.patch
        331 kB
        Per Steffensen

        Issue Links

          Activity

          Hide
          Per Steffensen added a comment -

          We are working on a solution that we hope to contribute soon.

          Show
          Per Steffensen added a comment - We are working on a solution that we hope to contribute soon.
          Hide
          Per Steffensen added a comment -

          Early patch attached to show details about suggested implementation and to get feedback.

          The patch should fit on top of your revision 1327417. The branch for doing the code for this issue, was created from the branch where we did the SOLR-3173, SOLR-3178 and SOLR-3382 changes, so it contains parts of an old fix for those. Some of those parts have been reverted or changed in the latest patch attached to SOLR-3178, so they are basically outdated. Hope you are able to distill what is related to SOLR-3383 and what is not.

          Basically the idea is that all public methods for doing requests using SolrJ SolrServer's or SolrRequest-subclasses, has been changed from returning responses (in the general form of NamedList<Object> or in the specialized form of SolrResponse-subclasses) into returning Futures of responses.

          Exceptions originating from the server, that you used to get when you called one of those public methods, you will now not get before you call get on the returned Future.

          The benefits are that you (if your chain of SolrServers contains a real async implementation) as a client, can go do other stuff from the time you send the request (by calling one of those public methods) and until the time you inspect the response (fetched by calling get on the returned Future). Besides that we will be able to get responses back from ConcurrentSolrServer.

          As is right now you need to have ConcurrentSolrServer in your chain of SolrServers in order to get real async behaviour, because the rest of the SolrServers does not really support it. If you dont use ConcurrentSolrServer your client thread will basically not get its Future back from a call to one of those public methods mentioned at the top, before the response has been received from the server and calling get on the Future will immediately give the client the reponse anyway. So not much acheived. But that can all easily be changed by modifying the HttpSolrServer to be able to run in a async mode where it has its own threadpool in which it does requests. That can easily be coded, and will to a big extend eliminate the need for a ConcurrentSolrServer to begin with. We will make this feature of the HttpSolrServer, and attach it if you like the idea represented by this issue SOLR-3383.

          The patch is a little bit dependent on the patch attached to SOLR-3178, in the way that it deals with e.g. PartialErrors in places where it needs to, but we can deal with that later.

          This is just to present the idea in code and give you something to look at and comment on.

          Regards, Per Steffensen

          Show
          Per Steffensen added a comment - Early patch attached to show details about suggested implementation and to get feedback. The patch should fit on top of your revision 1327417. The branch for doing the code for this issue, was created from the branch where we did the SOLR-3173 , SOLR-3178 and SOLR-3382 changes, so it contains parts of an old fix for those. Some of those parts have been reverted or changed in the latest patch attached to SOLR-3178 , so they are basically outdated. Hope you are able to distill what is related to SOLR-3383 and what is not. Basically the idea is that all public methods for doing requests using SolrJ SolrServer's or SolrRequest-subclasses, has been changed from returning responses (in the general form of NamedList<Object> or in the specialized form of SolrResponse-subclasses) into returning Futures of responses. Exceptions originating from the server, that you used to get when you called one of those public methods, you will now not get before you call get on the returned Future. The benefits are that you (if your chain of SolrServers contains a real async implementation) as a client, can go do other stuff from the time you send the request (by calling one of those public methods) and until the time you inspect the response (fetched by calling get on the returned Future). Besides that we will be able to get responses back from ConcurrentSolrServer. As is right now you need to have ConcurrentSolrServer in your chain of SolrServers in order to get real async behaviour, because the rest of the SolrServers does not really support it. If you dont use ConcurrentSolrServer your client thread will basically not get its Future back from a call to one of those public methods mentioned at the top, before the response has been received from the server and calling get on the Future will immediately give the client the reponse anyway. So not much acheived. But that can all easily be changed by modifying the HttpSolrServer to be able to run in a async mode where it has its own threadpool in which it does requests. That can easily be coded, and will to a big extend eliminate the need for a ConcurrentSolrServer to begin with. We will make this feature of the HttpSolrServer, and attach it if you like the idea represented by this issue SOLR-3383 . The patch is a little bit dependent on the patch attached to SOLR-3178 , in the way that it deals with e.g. PartialErrors in places where it needs to, but we can deal with that later. This is just to present the idea in code and give you something to look at and comment on. Regards, Per Steffensen
          Hide
          Hoss Man added a comment -

          bulk fixing the version info for 4.0-ALPHA and 4.0 all affected issues have "hoss20120711-bulk-40-change" in comment

          Show
          Hoss Man added a comment - bulk fixing the version info for 4.0-ALPHA and 4.0 all affected issues have "hoss20120711-bulk-40-change" in comment
          Hide
          Robert Muir added a comment -

          rmuir20120906-bulk-40-change

          Show
          Robert Muir added a comment - rmuir20120906-bulk-40-change
          Hide
          Robert Muir added a comment -

          moving all 4.0 issues not touched in a month to 4.1

          Show
          Robert Muir added a comment - moving all 4.0 issues not touched in a month to 4.1
          Hide
          Steve Rowe added a comment -

          Bulk move 4.4 issues to 4.5 and 5.0

          Show
          Steve Rowe added a comment - Bulk move 4.4 issues to 4.5 and 5.0
          Hide
          Martin Grotzke added a comment -

          I'm also interested in async responses in solrj, but instead of using java Futures, I'd prefer a callback based interface, e.g. via onSuccess(callback) and onError(callback), much like scala Futures work.

          Show
          Martin Grotzke added a comment - I'm also interested in async responses in solrj, but instead of using java Futures, I'd prefer a callback based interface, e.g. via onSuccess(callback) and onError(callback), much like scala Futures work.
          Hide
          Per Steffensen added a comment -

          Martin Grotzke, we made a java-Futures implementation long ago. We have it on some branch somewhere. We never got around to finish it, because we decided that it was not high priority at the moment. We will not prioritize to finish it any time soon. Callback thingy could also be interesting - maybe support both java-Futures and callback. Guess if you want this, you will have to make it yourself. Guess I can give you a patch with our changes, but it will fit a very old version of Solr - probably somewhere before 4.0.0 was even released. You will definitely have to work on it afterwards.

          Show
          Per Steffensen added a comment - Martin Grotzke , we made a java-Futures implementation long ago. We have it on some branch somewhere. We never got around to finish it, because we decided that it was not high priority at the moment. We will not prioritize to finish it any time soon. Callback thingy could also be interesting - maybe support both java-Futures and callback. Guess if you want this, you will have to make it yourself. Guess I can give you a patch with our changes, but it will fit a very old version of Solr - probably somewhere before 4.0.0 was even released. You will definitely have to work on it afterwards.
          Hide
          Martin Grotzke added a comment -

          Ok, thanks for the update.

          Show
          Martin Grotzke added a comment - Ok, thanks for the update.
          Hide
          Uwe Schindler added a comment -

          Move issue to Solr 4.9.

          Show
          Uwe Schindler added a comment - Move issue to Solr 4.9.

            People

            • Assignee:
              Per Steffensen
              Reporter:
              Per Steffensen
            • Votes:
              3 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:

                Development