Solr
  1. Solr
  2. SOLR-2458

post.jar fails on non-XML updateHandlers

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.4
    • Component/s: None
    • Labels:

      Description

      SimplePostTool.java by default tries to issue a commit after posting.
      Problem is that it does this by appending "<commit/>" to the stream.
      This does not work when using non-XML requesthandler, such as CSV.

      1. SOLR-2458.patch
        4 kB
        Jan Høydahl
      2. SOLR-2458.patch
        5 kB
        Jan Høydahl
      3. SOLR-2458.patch
        6 kB
        Jan Høydahl

        Activity

        Hide
        Jan Høydahl added a comment -

        Example:

        lap:exampledocs janhoy$ java -Durl=http://localhost:8983/solr/update/csv -jar post.jar books.csv
        SimplePostTool: version 1.3
        SimplePostTool: POSTing files to http://localhost:8983/solr/update/csv..
        SimplePostTool: POSTing file books.csv
        SimplePostTool: COMMITting Solr index changes..
        SimplePostTool: FATAL: Solr returned an error #400 undefined field <commit/>
        

        The commit should be sent in a different way, problem is how to know where and how to send the commit in case of non-standard URLs, such as http://localhost:8983/solr/my/custom/updatehandler

        Show
        Jan Høydahl added a comment - Example: lap:exampledocs janhoy$ java -Durl=http: //localhost:8983/solr/update/csv -jar post.jar books.csv SimplePostTool: version 1.3 SimplePostTool: POSTing files to http: //localhost:8983/solr/update/csv.. SimplePostTool: POSTing file books.csv SimplePostTool: COMMITting Solr index changes.. SimplePostTool: FATAL: Solr returned an error #400 undefined field <commit/> The commit should be sent in a different way, problem is how to know where and how to send the commit in case of non-standard URLs, such as http://localhost:8983/solr/my/custom/updatehandler
        Hide
        Uwe Schindler added a comment -

        The commit could be snet at the end as a single xml document in a separate request if the content type of data is different.

        Show
        Uwe Schindler added a comment - The commit could be snet at the end as a single xml document in a separate request if the content type of data is different.
        Hide
        Jan Høydahl added a comment -

        How would post.jar know the URL of the XmlUpdateRequestHandler?
        A) We could assume .*/solr/update as 99% would not modify the defaults?
        Or
        B) Assume that all UpdateRequestHandlers support a GET parameter commit=true
        In that case, we could append "?commit=true" to the given URL.
        I know for a fact that "/solr/update/csv?commit=true" will work

        Show
        Jan Høydahl added a comment - How would post.jar know the URL of the XmlUpdateRequestHandler? A) We could assume .*/solr/update as 99% would not modify the defaults? Or B) Assume that all UpdateRequestHandlers support a GET parameter commit=true In that case, we could append "?commit=true" to the given URL. I know for a fact that "/solr/update/csv?commit=true" will work
        Hide
        Yonik Seeley added a comment -

        Assume that all UpdateRequestHandlers support a GET parameter commit=true

        I think we should assume this, and fix anything where it doesn't work.

        Show
        Yonik Seeley added a comment - Assume that all UpdateRequestHandlers support a GET parameter commit=true I think we should assume this, and fix anything where it doesn't work.
        Hide
        Hoss Man added a comment -

        post.jar has hardcoded assumptions about what URL you want to hit and how it should behave – if you want to change those assumptions there are documented params for changing it. -Durl=... and -Dcommit=false.

        if you want to post to something that isn't the XmlRequestHandler, you should specify -Dcommit=false, and then you can follow that with an explicit execution to commit...

        java -Durl=... -jar post.jar *.cssv
        java -jar post.jar

        part of the reason it works the way it does now is that when commit=true it POSTs a single commit at the end of multiple file POSTs, if we use the param based commit it would either need to specify commit on all of them, or keep track of the last one only add the param there.

        i don't object to changing post.jar to use a commit request param instead of sending the XML form, but this isn't a bug – it's working as it was intended.

        Show
        Hoss Man added a comment - post.jar has hardcoded assumptions about what URL you want to hit and how it should behave – if you want to change those assumptions there are documented params for changing it. -Durl=... and -Dcommit=false. if you want to post to something that isn't the XmlRequestHandler, you should specify -Dcommit=false, and then you can follow that with an explicit execution to commit... java -Durl=... -jar post.jar *.cssv java -jar post.jar part of the reason it works the way it does now is that when commit=true it POSTs a single commit at the end of multiple file POSTs, if we use the param based commit it would either need to specify commit on all of them, or keep track of the last one only add the param there. i don't object to changing post.jar to use a commit request param instead of sending the XML form, but this isn't a bug – it's working as it was intended.
        Hide
        Yonik Seeley added a comment - - edited

        part of the reason it works the way it does now is that when commit=true it POSTs a single commit at the end of multiple file POSTs, if we use the param based commit it would either need to specify commit on all of them, or keep track of the last one only add the param there.

        Although only adding a "commit" on the last update should be easy, we could also just do it via the URL. I believe posting "?commit=true" to update handlers w/o a body works?

        Show
        Yonik Seeley added a comment - - edited part of the reason it works the way it does now is that when commit=true it POSTs a single commit at the end of multiple file POSTs, if we use the param based commit it would either need to specify commit on all of them, or keep track of the last one only add the param there. Although only adding a "commit" on the last update should be easy, we could also just do it via the URL. I believe posting "?commit=true" to update handlers w/o a body works?
        Hide
        Jan Høydahl added a comment -

        It might not be a bug according to the original design intentions. But the first thing we tell users is to try out post.jar to post stuff, and now we've even included csv and json examples for it. Then it's unnecessary to get the error "Solr returned an error #400 undefined field <commit/>" thrown in your face - the error does not even explain the problem.

        I'll try to assemble a first patch for this next week some time, adding a separate POST with ?commit=true after the last file is POSTed.

        Show
        Jan Høydahl added a comment - It might not be a bug according to the original design intentions. But the first thing we tell users is to try out post.jar to post stuff, and now we've even included csv and json examples for it. Then it's unnecessary to get the error "Solr returned an error #400 undefined field <commit/>" thrown in your face - the error does not even explain the problem. I'll try to assemble a first patch for this next week some time, adding a separate POST with ?commit=true after the last file is POSTed.
        Hide
        Jan Høydahl added a comment -

        The solution was simple. Change the commit() method to do ?commit=true instead of posting as "<commit/>"

        Also cleaned up dead meat, added a -Doptimize=yes option and accepts -h and --help in addition to -help

        Show
        Jan Høydahl added a comment - The solution was simple. Change the commit() method to do ?commit=true instead of posting as "<commit/>" Also cleaned up dead meat, added a -Doptimize=yes option and accepts -h and --help in addition to -help
        Hide
        Jan Høydahl added a comment -

        This new patch bumps version number to 1.4 and adds examples to the help of how to post csv, json and pdf

        Show
        Jan Høydahl added a comment - This new patch bumps version number to 1.4 and adds examples to the help of how to post csv, json and pdf
        Hide
        Jan Høydahl added a comment - - edited

        Anyone fancy testing this and consider for commit?

        Show
        Jan Høydahl added a comment - - edited Anyone fancy testing this and consider for commit?
        Hide
        Mark Miller added a comment -

        Hey Jan - putting this on my to-do list in the case that someone does not beat me to it. You have a variety of issues out there that I'd like to get to I think - sorry for the delay. You have been a fantastic addition to the project.

        Show
        Mark Miller added a comment - Hey Jan - putting this on my to-do list in the case that someone does not beat me to it. You have a variety of issues out there that I'd like to get to I think - sorry for the delay. You have been a fantastic addition to the project.
        Hide
        Jan Høydahl added a comment -

        Marking fo 3.2 since patch is ready

        Show
        Jan Høydahl added a comment - Marking fo 3.2 since patch is ready
        Hide
        Robert Muir added a comment -

        Bulk move 3.2 -> 3.3

        Show
        Robert Muir added a comment - Bulk move 3.2 -> 3.3
        Hide
        Jan Høydahl added a comment -

        Has anyone got around to inspecting this patch? I'd like to get this into 3.3.

        Show
        Jan Høydahl added a comment - Has anyone got around to inspecting this patch? I'd like to get this into 3.3.
        Hide
        Hoss Man added a comment -

        Jan: +1

        Show
        Hoss Man added a comment - Jan: +1
        Hide
        Jan Høydahl added a comment -

        Attaching final patch which will be committed shortly. Added better error handling.

        Show
        Jan Høydahl added a comment - Attaching final patch which will be committed shortly. Added better error handling.
        Hide
        Jan Høydahl added a comment -

        Committed for trunk and 3.x

        Show
        Jan Høydahl added a comment - Committed for trunk and 3.x
        Hide
        Robert Muir added a comment -

        bulk close for 3.4

        Show
        Robert Muir added a comment - bulk close for 3.4

          People

          • Assignee:
            Jan Høydahl
            Reporter:
            Jan Høydahl
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development