Solr
  1. Solr
  2. SOLR-3382

Finegrained error propagation (focus on multi-document updates)

    Details

      Description

      Today when an error occurs on server side during the handling of a request, the handling of the request will stop at the point when the error occured, and only a error-message (reason part of HTTP status line) and error-code (HTTP response code) is pushed back to the sender of the request.

      This can be improved in several ways

      • Reacting as a client on errors, solely based on a textual message and a HTTP response code is hard. The error ought have some kind of type telling the client which kind of error happened.
      • When handling multi-document updates the error might happen during the handling of one of the documents - potentially not the first document and potentially not the last document.
        • The client ought to get information about which documents where successfully updated (the ones comming before the document creating the error).
        • If the error updating a document is not due to a general problem, it could very well be that some of the documents not yet handled at the time of the error (the documents comming after the document creating the error), could be successfully updated - so why not try that.
        • If continuing the updating of documents, even after one document-update resulted in an error (as suggested above), the updating of some of those documents might also result in an error. This leads to another improvement, namely being able to send information about more than one error back to the client.

        Issue Links

          Activity

          Hide
          Per Steffensen added a comment -

          We have created a solution to this while adding the update-features of "fail if intent is to insert new document and document already exists", "fail if intent is to update existing document and document does not exist or has been changed since loaded" etc. as described in SOLR-3173, SOLR-3178 and on http://wiki.apache.org/solr/Per%20Steffensen/Update%20semantics. Details about the solution to this SOLR-3382 is also to be found on http://wiki.apache.org/solr/Per%20Steffensen/Update%20semantics. Reason for solving this SOLR-3382 at the same time as SOLR-3173 and SOLR-3178 is that "unique key constraint"- and "versioning"-errors are not worth much if they cannot get properly propagated to the client.

          We hope to contribe this solutions within a few weeks, but please feel free to comment, help shape etc., and we will make changes according to the wishes of the community.

          Show
          Per Steffensen added a comment - We have created a solution to this while adding the update-features of "fail if intent is to insert new document and document already exists", "fail if intent is to update existing document and document does not exist or has been changed since loaded" etc. as described in SOLR-3173 , SOLR-3178 and on http://wiki.apache.org/solr/Per%20Steffensen/Update%20semantics . Details about the solution to this SOLR-3382 is also to be found on http://wiki.apache.org/solr/Per%20Steffensen/Update%20semantics . Reason for solving this SOLR-3382 at the same time as SOLR-3173 and SOLR-3178 is that "unique key constraint"- and "versioning"-errors are not worth much if they cannot get properly propagated to the client. We hope to contribe this solutions within a few weeks, but please feel free to comment, help shape etc., and we will make changes according to the wishes of the community.
          Hide
          Per Steffensen added a comment -

          In our contribution the "finegrained error propagation" is solely used for reporting "unique key constraint"- and "versioning"-errors, but the design is very generic, so in the future it can also be used for other situations where you want to report "partial errors" back to the client - e.g. in a replication setup, information about a update succeeding on leader but not on one or more of the replica.

          Show
          Per Steffensen added a comment - In our contribution the "finegrained error propagation" is solely used for reporting "unique key constraint"- and "versioning"-errors, but the design is very generic, so in the future it can also be used for other situations where you want to report "partial errors" back to the client - e.g. in a replication setup, information about a update succeeding on leader but not on one or more of the replica.
          Hide
          Stefan Matheis (steffkes) added a comment -

          Per, just one quick thought, regarding http://wiki.apache.org/solr/Per%20Steffensen/Update%20semantics#Status_line :
          Can we stick with default reason-phrases and put ours in X-Error-* -Headers? Otherwise we have to put String-Operations in Place, if we want to output these informations in some type of frontend

          Show
          Stefan Matheis (steffkes) added a comment - Per, just one quick thought, regarding http://wiki.apache.org/solr/Per%20Steffensen/Update%20semantics#Status_line : Can we stick with default reason-phrases and put ours in X-Error-* -Headers? Otherwise we have to put String-Operations in Place, if we want to output these informations in some type of frontend
          Hide
          Per Steffensen added a comment -

          Stefan Metheis, the string-operations are already there for you if you are using SolrJ, but I guess you are not, since you (as I understand it) are dealing with Solr admin console (probably calling services like update/query directly from the browser using some kind of Ajax, instead of doing that on server, which would be natural if the admin console where e.g. JSP-based). But never mind, your idea about encoding error-type (namespace and name) in response headers instead of in the reason-phrase is better. I will make the necessary changes in our code, so that it will be part of the contribution, and change the Wiki page. Thanks a lot!

          Show
          Per Steffensen added a comment - Stefan Metheis, the string-operations are already there for you if you are using SolrJ, but I guess you are not, since you (as I understand it) are dealing with Solr admin console (probably calling services like update/query directly from the browser using some kind of Ajax, instead of doing that on server, which would be natural if the admin console where e.g. JSP-based). But never mind, your idea about encoding error-type (namespace and name) in response headers instead of in the reason-phrase is better. I will make the necessary changes in our code, so that it will be part of the contribution, and change the Wiki page. Thanks a lot!
          Hide
          Per Steffensen added a comment -

          Stefan: Done! Changed my code and Wiki page. Please review and comment if you are not satisfied.

          Show
          Per Steffensen added a comment - Stefan: Done! Changed my code and Wiki page. Please review and comment if you are not satisfied.
          Hide
          Per Steffensen added a comment -

          See patch attached to SOLR-3178

          Show
          Per Steffensen added a comment - See patch attached to SOLR-3178
          Hide
          Per Steffensen added a comment -

          New patch available as part of SOLR-3173_3178_3382_3428_plus.patch on SOLR-3178

          Show
          Per Steffensen added a comment - New patch available as part of SOLR-3173 _3178_3382_3428_plus.patch on SOLR-3178
          Hide
          Hoss Man added a comment -

          bulk fixing the version info for 4.0-ALPHA and 4.0 all affected issues have "hoss20120711-bulk-40-change" in comment

          Show
          Hoss Man added a comment - bulk fixing the version info for 4.0-ALPHA and 4.0 all affected issues have "hoss20120711-bulk-40-change" in comment
          Hide
          Robert Muir added a comment -

          rmuir20120906-bulk-40-change

          Show
          Robert Muir added a comment - rmuir20120906-bulk-40-change
          Hide
          Robert Muir added a comment -

          moving all 4.0 issues not touched in a month to 4.1

          Show
          Robert Muir added a comment - moving all 4.0 issues not touched in a month to 4.1
          Hide
          Per Steffensen added a comment - - edited

          We use this Apache issue/ticket to keep track of what we did on our own internal version of Solr, and what would be provided in a patch for this issue if the Apache community was interested. Therefore this additional comment, for now just a "note to self":

          The solution for "error propagation" now not only supports sending error-code, error-msg and error-type (corresponding to package+classname in java) so that exceptions can be reconstructed and thrown on client-side (automatic in SolrJ). Now it also supports additional properties (name/value pairs) on an exception and have those propagated over the line to the client where it can be (will be in SolrJ) attached as properties on the reconstructed exception. Explicit properties on exception are nice, because only alternative is to "encode" it in the error-msg, and then you have to parse the error-msg on the receiving side and that is part of what we want to avoid (it is too hacky and error prone).
          Properties also supported on "nested" exceptions about parts of the request-handling that failed - e.g. in a 5-doc update-request, where 3 updates succeed and 2 fail, 2 "nested/part"-exceptions will be returned to the client. Both of those "nested/part"-exceptions can independently contain their own set of properties.

          As a proof-of-concept and to show that the implementation works and to have the feature "protected" by tests, the VersionConflict exception introduced in the solution for SOLR-3178 has been enhanced to carry a property "currentVersion" containing the current version of the document on server-side. So now, if the client want to know about the actual current version on server-side, it does not have to parse the error-msg to get hold of it - just read the provided property. All existing tests using VersionConflict has been updated to test that the returned currentVersion is correct. There are test that have VersionConflict as the top-level exception (version-conflict in a single-doc update-request) and other tests that have several VersionConflicts as "nested/part"-exceptions so both aspects are tested.

          The ability of VersionConflict to carry current-version might not be very usefull. But it shows that the feature implementation works and it makes it easy to protect the feature by tests. Internally we will be using the feature for somthing far more usefull.

          Show
          Per Steffensen added a comment - - edited We use this Apache issue/ticket to keep track of what we did on our own internal version of Solr, and what would be provided in a patch for this issue if the Apache community was interested. Therefore this additional comment, for now just a "note to self": The solution for "error propagation" now not only supports sending error-code, error-msg and error-type (corresponding to package+classname in java) so that exceptions can be reconstructed and thrown on client-side (automatic in SolrJ). Now it also supports additional properties (name/value pairs) on an exception and have those propagated over the line to the client where it can be (will be in SolrJ) attached as properties on the reconstructed exception. Explicit properties on exception are nice, because only alternative is to "encode" it in the error-msg, and then you have to parse the error-msg on the receiving side and that is part of what we want to avoid (it is too hacky and error prone). Properties also supported on "nested" exceptions about parts of the request-handling that failed - e.g. in a 5-doc update-request, where 3 updates succeed and 2 fail, 2 "nested/part"-exceptions will be returned to the client. Both of those "nested/part"-exceptions can independently contain their own set of properties. As a proof-of-concept and to show that the implementation works and to have the feature "protected" by tests, the VersionConflict exception introduced in the solution for SOLR-3178 has been enhanced to carry a property "currentVersion" containing the current version of the document on server-side. So now, if the client want to know about the actual current version on server-side, it does not have to parse the error-msg to get hold of it - just read the provided property. All existing tests using VersionConflict has been updated to test that the returned currentVersion is correct. There are test that have VersionConflict as the top-level exception (version-conflict in a single-doc update-request) and other tests that have several VersionConflicts as "nested/part"-exceptions so both aspects are tested. The ability of VersionConflict to carry current-version might not be very usefull. But it shows that the feature implementation works and it makes it easy to protect the feature by tests. Internally we will be using the feature for somthing far more usefull.
          Hide
          Mark Miller added a comment -

          I'd still like to get this piece in somehow - I'm not sure how back compat handled (or much of anything - I have not looked closely yet), but even if it just goes into 5.0 instead, I think this is really important to get done.

          Show
          Mark Miller added a comment - I'd still like to get this piece in somehow - I'm not sure how back compat handled (or much of anything - I have not looked closely yet), but even if it just goes into 5.0 instead, I think this is really important to get done.
          Hide
          Steve Rowe added a comment -

          Bulk move 4.4 issues to 4.5 and 5.0

          Show
          Steve Rowe added a comment - Bulk move 4.4 issues to 4.5 and 5.0
          Hide
          Uwe Schindler added a comment -

          Move issue to Solr 4.9.

          Show
          Uwe Schindler added a comment - Move issue to Solr 4.9.

            People

            • Assignee:
              Per Steffensen
              Reporter:
              Per Steffensen
            • Votes:
              4 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:

                Development