Solr
  1. Solr
  2. SOLR-86

[PATCH] standalone updater cli based on httpClient

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.2
    • Component/s: update
    • Labels:
      None

      Description

      We need a cross platform replacement for the post.sh.
      The attached code is a direct replacement of the post.sh since it is actually doing the same exact thing.

      In the future one can extend the CLI with other feature like auto commit, etc..
      Right now the code assumes that SOLR-85 is applied since we using the servlet of this issue to actually do the update.

      1. solr-86.diff
        11 kB
      2. solr-86.diff
        13 kB
        Thorsten Scherler
      3. simple-post-using-urlconnection-approach.patch
        6 kB
        Hoss Man
      4. simple-post-tool-2007-02-15.patch
        7 kB
        Bertrand Delacretaz
      5. simple-post-tool-2007-02-16.patch
        8 kB
        Bertrand Delacretaz

        Issue Links

          Activity

          Hide
          Thorsten Scherler added a comment -

          Index: src/tools/updater/lib/commons-httpclient-3.0.1.jar
          Index: src/tools/updater/lib/commons-codec-1.3.jar
          Index: src/tools/updater/lib/commons-logging-1.1.jar
          see http://jakarta.apache.org/commons/httpclient/dependencies.html for the download pages.

          After applying the patch and adding above libs to src/tools/updater/lib you can install the updater with:
          ant updater.jar
          and follow the instructions given in this target.

          Show
          Thorsten Scherler added a comment - Index: src/tools/updater/lib/commons-httpclient-3.0.1.jar Index: src/tools/updater/lib/commons-codec-1.3.jar Index: src/tools/updater/lib/commons-logging-1.1.jar see http://jakarta.apache.org/commons/httpclient/dependencies.html for the download pages. After applying the patch and adding above libs to src/tools/updater/lib you can install the updater with: ant updater.jar and follow the instructions given in this target.
          Hide
          Thorsten Scherler added a comment -

          Adding auto commit to the updater.

          Show
          Thorsten Scherler added a comment - Adding auto commit to the updater.
          Hide
          Otis Gospodnetic added a comment -

          Are you working on unifying SOLR-20, SOLR-30, and this into once coherent java client package? I think that would make quite a few people happy.

          Show
          Otis Gospodnetic added a comment - Are you working on unifying SOLR-20 , SOLR-30 , and this into once coherent java client package? I think that would make quite a few people happy.
          Hide
          Hoss Man added a comment -

          Independent of the work being done in SOLR-20, i think it would be very usefull to have a pure java replacement for post.sh – both so people on machines wihout bash/curl can try out the simple tutorial, and as a bare bones example of how to do a simple POST to Solr in java.

          Thorsten: depending on SOLR-85 seems unrelated to the goal of "a cross platform replacement for the post.sh" ... I'm also not convinced this is really a use case where depending on HttpClient (and all it requires) really makes sense ... if the goal is a simple demonstrative tool then it should have as few dependencies as possible right?

          I've been playing arround with your attachment a bit and i've got an alternate version i'd like your feedback on ... for simplicity i left the code in the util package of the main code tree, and modified the main build.xml so that "ant example" would create example/post.jar used like so...

          java -jar example/post.jar http://localhost:8983/solr/update example/exampledocs/*.xml

          attachment to follow.

          Show
          Hoss Man added a comment - Independent of the work being done in SOLR-20 , i think it would be very usefull to have a pure java replacement for post.sh – both so people on machines wihout bash/curl can try out the simple tutorial, and as a bare bones example of how to do a simple POST to Solr in java. Thorsten: depending on SOLR-85 seems unrelated to the goal of "a cross platform replacement for the post.sh" ... I'm also not convinced this is really a use case where depending on HttpClient (and all it requires) really makes sense ... if the goal is a simple demonstrative tool then it should have as few dependencies as possible right? I've been playing arround with your attachment a bit and i've got an alternate version i'd like your feedback on ... for simplicity i left the code in the util package of the main code tree, and modified the main build.xml so that "ant example" would create example/post.jar used like so... java -jar example/post.jar http://localhost:8983/solr/update example/exampledocs/*.xml attachment to follow.
          Hide
          Bertrand Delacretaz added a comment -

          I like the idea of a very simple "update only" client.

          It's probably simple enough to create two versions, one using HttpClient and one with no dependencies apart from the JDK? I agree with Hoss that the post.sh replacement should use the latter.

          IMHO it's good to show the use of HttpClient for people who're going to base more complex clients on it, and a "no depedencies" client is useful for simple cases.

          Maybe (thinking outloud here) both clients could implement a common SolrUpdateClientInterface, and update+search clients would implement a SolrSearchInterface as well.

          Show
          Bertrand Delacretaz added a comment - I like the idea of a very simple "update only" client. It's probably simple enough to create two versions, one using HttpClient and one with no dependencies apart from the JDK? I agree with Hoss that the post.sh replacement should use the latter. IMHO it's good to show the use of HttpClient for people who're going to base more complex clients on it, and a "no depedencies" client is useful for simple cases. Maybe (thinking outloud here) both clients could implement a common SolrUpdateClientInterface, and update+search clients would implement a SolrSearchInterface as well.
          Hide
          Thorsten Scherler added a comment -

          Hi Hoiss, I had a look at your version and it is good as gold.
          I personally prefer the httpClient since the method is smaller but Bertrand and ourself are right, the dependency jar price for a simple replacement is ATM too high.

          The only thing that I would add is directory support:
          ...
          + if (srcFile.exists()) {
          + if (srcFile.isDirectory()) {
          + File[] fileSet = srcFile.listFiles();
          + for (int i = 0; i < fileSet.length; i++)

          { + File file = fileSet[i]; + tool.postFile(file, out); + }

          else

          { + tool.postFile(srcFile, out); + }

          + System.out.println();
          + } else

          { + System.err.println(srcFile + " does not exist"); + }

          I agree to your patch as official replacement of the post.sh. I further agree with Bertrand that we may include patch as base demonstration for more complex client apps.

          Show
          Thorsten Scherler added a comment - Hi Hoiss, I had a look at your version and it is good as gold. I personally prefer the httpClient since the method is smaller but Bertrand and ourself are right, the dependency jar price for a simple replacement is ATM too high. The only thing that I would add is directory support: ... + if (srcFile.exists()) { + if (srcFile.isDirectory()) { + File[] fileSet = srcFile.listFiles(); + for (int i = 0; i < fileSet.length; i++) { + File file = fileSet[i]; + tool.postFile(file, out); + } else { + tool.postFile(srcFile, out); + } + System.out.println(); + } else { + System.err.println(srcFile + " does not exist"); + } I agree to your patch as official replacement of the post.sh. I further agree with Bertrand that we may include patch as base demonstration for more complex client apps.
          Hide
          Hoss Man added a comment -

          regarding Bertrand's comment, i'm not sure if there is any benefit in having this code and SOLR-20 share a common "SolrUpdateClientInterface" since this code will be dealing with pure streaming of UTF8 data, while SOLR-20 is focused on a better object abstracting for SolrDocuments ... i'm not sure what kinds of methods such an interface might have.

          regarding THorstens comment: yeah, i removed the directory support from your patch while i was refactoring just because it was confusing me and i was trying to keep things simple (i kept trying to run java -jar post.jar exampledos/ ant it would fail because of the .svn directory)

          that's no reason not to inlcude it though since it's so simple.

          Show
          Hoss Man added a comment - regarding Bertrand's comment, i'm not sure if there is any benefit in having this code and SOLR-20 share a common "SolrUpdateClientInterface" since this code will be dealing with pure streaming of UTF8 data, while SOLR-20 is focused on a better object abstracting for SolrDocuments ... i'm not sure what kinds of methods such an interface might have. regarding THorstens comment: yeah, i removed the directory support from your patch while i was refactoring just because it was confusing me and i was trying to keep things simple (i kept trying to run java -jar post.jar exampledos/ ant it would fail because of the .svn directory) that's no reason not to inlcude it though since it's so simple.
          Hide
          Thorsten Scherler added a comment -

          Yeah, I know what you mean (had a similar problem today).

          if (!file.isDirectory())

          { tool.postFile(file, out); }

          should fix that.

          TIA

          Show
          Thorsten Scherler added a comment - Yeah, I know what you mean (had a similar problem today). if (!file.isDirectory()) { tool.postFile(file, out); } should fix that. TIA
          Hide
          Bertrand Delacretaz added a comment -

          I have reworked the SimplePostTool, improved the error reporting (less verbose but still informative enough I think), and added example parameters to make it self-explaining.

          I've also changed build.xml, the post.jar now goes to example/exampledocs, the same place as post.sh.

          Like the original SimplePostTool, this does not require any additional Java library. I think it's good enough to be a portable replacement for post.sh.

          Show
          Bertrand Delacretaz added a comment - I have reworked the SimplePostTool, improved the error reporting (less verbose but still informative enough I think), and added example parameters to make it self-explaining. I've also changed build.xml, the post.jar now goes to example/exampledocs, the same place as post.sh. Like the original SimplePostTool, this does not require any additional Java library. I think it's good enough to be a portable replacement for post.sh.
          Hide
          Bertrand Delacretaz added a comment -

          Here's another update of SimplePostTool (simple-post-tool-2007-02-16.patch) with an improved "user interface".

          In the happy case, the tool reports what it's doing in detail (yes, we could add a "be quiet" switch , checking Solr's responses to decide whether POSTing went well:

          $ java -jar post.jar http://localhost:8983/solr/update *.xml
          SimplePostTool: $Id$
          SimplePostTool: WARNING: Make sure your XML documents are encoded in UTF-8, other encodings are not currently supported
          SimplePostTool: POSTing files to http://localhost:8983/solr/update..
          SimplePostTool: POSTing file hd.xml
          SimplePostTool: POSTing file utf8-example.xml
          SimplePostTool: COMMITting Solr index changes..
          SimplePostTool: 2 files POSTed to http://localhost:8983/solr/update

          $Id$ will be the SVN revision.

          Solr responses are checked as Strings, this should be improved by parsing them as XML.

          If the Solr URL does not respond, the error message should help:

          $ java -jar post.jar http://localhost:8983/solr/update *.xml
          SimplePostTool: $Id$
          SimplePostTool: WARNING: Make sure your XML documents are encoded in UTF-8, other encodings are not currently supported
          SimplePostTool: POSTing files to http://localhost:8983/solr/update..
          SimplePostTool: POSTing file hd.xml
          SimplePostTool: FATAL: Connection error (is Solr running at http://localhost:8983/solr/update ?): java.net.ConnectException: Connection refused

          Show
          Bertrand Delacretaz added a comment - Here's another update of SimplePostTool (simple-post-tool-2007-02-16.patch) with an improved "user interface". In the happy case, the tool reports what it's doing in detail (yes, we could add a "be quiet" switch , checking Solr's responses to decide whether POSTing went well: $ java -jar post.jar http://localhost:8983/solr/update *.xml SimplePostTool: $Id$ SimplePostTool: WARNING: Make sure your XML documents are encoded in UTF-8, other encodings are not currently supported SimplePostTool: POSTing files to http://localhost:8983/solr/update .. SimplePostTool: POSTing file hd.xml SimplePostTool: POSTing file utf8-example.xml SimplePostTool: COMMITting Solr index changes.. SimplePostTool: 2 files POSTed to http://localhost:8983/solr/update $Id$ will be the SVN revision. Solr responses are checked as Strings, this should be improved by parsing them as XML. If the Solr URL does not respond, the error message should help: $ java -jar post.jar http://localhost:8983/solr/update *.xml SimplePostTool: $Id$ SimplePostTool: WARNING: Make sure your XML documents are encoded in UTF-8, other encodings are not currently supported SimplePostTool: POSTing files to http://localhost:8983/solr/update .. SimplePostTool: POSTing file hd.xml SimplePostTool: FATAL: Connection error (is Solr running at http://localhost:8983/solr/update ?): java.net.ConnectException: Connection refused
          Hide
          Erik Hatcher added a comment -

          Wow, so much great stuff happening in the Solr system!

          I'd love to have this tool built-in for my Solr workshop coming up. Hoss, you game for committing this since you assigned it to yourself? Or if you want I'd be happy to add it.

          Show
          Erik Hatcher added a comment - Wow, so much great stuff happening in the Solr system! I'd love to have this tool built-in for my Solr workshop coming up. Hoss, you game for committing this since you assigned it to yourself? Or if you want I'd be happy to add it.
          Hide
          Erik Hatcher added a comment -

          Apologies to Hoss since this was assigned to him, but I couldn't resist committing this little gem. The tool works very well and removes the startup hassles Windows users have with Cygwin or other environments where curl may not be installed. All Solr users gotta hava Java!

          Show
          Erik Hatcher added a comment - Apologies to Hoss since this was assigned to him, but I couldn't resist committing this little gem. The tool works very well and removes the startup hassles Windows users have with Cygwin or other environments where curl may not be installed. All Solr users gotta hava Java!
          Hide
          Hoss Man added a comment -

          you guys are too quick for me... i planed on working on this this weekend, and before i get a chance Bertrand overhauls it and improves the error messages, and erik commits it.

          thanks again to Thorsten for getting the ball rolling on this.

          Show
          Hoss Man added a comment - you guys are too quick for me... i planed on working on this this weekend, and before i get a chance Bertrand overhauls it and improves the error messages, and erik commits it. thanks again to Thorsten for getting the ball rolling on this.
          Hide
          Bertrand Delacretaz added a comment -

          Considering the tutorial examples (http://lucene.apache.org/solr/tutorial.html), it'd be useful to allow this to POST its standard input, or the contents of a command-line parameter:

          Use case:

          echo "<delete><query>name:DDR</query></delete>" | java -jar post.jar http://localhost:8983

          or (having both would be useful I think):

          java -jar post.jar -data "<delete><query>name:DDR</query></delete>" http://localhost:8983

          Show
          Bertrand Delacretaz added a comment - Considering the tutorial examples ( http://lucene.apache.org/solr/tutorial.html ), it'd be useful to allow this to POST its standard input, or the contents of a command-line parameter: Use case: echo "<delete><query>name:DDR</query></delete>" | java -jar post.jar http://localhost:8983 or (having both would be useful I think): java -jar post.jar -data "<delete><query>name:DDR</query></delete>" http://localhost:8983
          Hide
          Hoss Man added a comment -

          yeah ... i think we should hardcode http://localhost:8983/solr/update with a possible override by system prop, then add either a command line switch other another system prop indicating to use the command line as filenames or as raw data, and another op for stdin.

          java -jar -Ddata=files post.jar *.xml
          java -jar post.jar *.xml ... data=files being the default
          echo "<delete><query>name:DDR</query></delete>" | java -jar -Ddata=stdin post.jar
          cat *.xml | java -jar -Ddata=stdin post.jar
          java -jar -Ddata=args post.jar "<delete><query>name:DDR</query></delete>"
          java -jar -Durl=http://localhost:8983/solr/update post.jar *.xml

          Show
          Hoss Man added a comment - yeah ... i think we should hardcode http://localhost:8983/solr/update with a possible override by system prop, then add either a command line switch other another system prop indicating to use the command line as filenames or as raw data, and another op for stdin. java -jar -Ddata=files post.jar *.xml java -jar post.jar *.xml ... data=files being the default echo "<delete><query>name:DDR</query></delete>" | java -jar -Ddata=stdin post.jar cat *.xml | java -jar -Ddata=stdin post.jar java -jar -Ddata=args post.jar "<delete><query>name:DDR</query></delete>" java -jar -Durl= http://localhost:8983/solr/update post.jar *.xml
          Hide
          Will Johnson added a comment -

          has anyone brought up the idea of creating post.bat and post.sh scripts that use this java class instead of the curl example that currently ships in example/exampledocs? it would be one less thing for people to figure out and possibly screw up.

          Show
          Will Johnson added a comment - has anyone brought up the idea of creating post.bat and post.sh scripts that use this java class instead of the curl example that currently ships in example/exampledocs? it would be one less thing for people to figure out and possibly screw up.
          Hide
          Hoss Man added a comment -

          this will ship in the next release, and the tutorial that will ship with that release already refers to it.

          creating a post.sh or post.bat that delegates to this tool seems like it can only complicate things ... file perms, line endings, shell conventions, shebang lines ... all things where portability is a concern, but "java -jar post.jar *.xml" works damn near anywhere.

          Show
          Hoss Man added a comment - this will ship in the next release, and the tutorial that will ship with that release already refers to it. creating a post.sh or post.bat that delegates to this tool seems like it can only complicate things ... file perms, line endings, shell conventions, shebang lines ... all things where portability is a concern, but "java -jar post.jar *.xml" works damn near anywhere.
          Hide
          Hoss Man added a comment -

          This bug was modified as part of a bulk update using the criteria...

          • Marked ("Resolved" or "Closed") and "Fixed"
          • Had no "Fix Version" versions
          • Was listed in the CHANGES.txt for 1.2

          The Fix Version for all 39 issues found was set to 1.2, email notification
          was suppressed to prevent excessive email.

          For a list of all the issues modified, search jira comments for this
          (hopefully) unique string: 20080415hossman2

          Show
          Hoss Man added a comment - This bug was modified as part of a bulk update using the criteria... Marked ("Resolved" or "Closed") and "Fixed" Had no "Fix Version" versions Was listed in the CHANGES.txt for 1.2 The Fix Version for all 39 issues found was set to 1.2, email notification was suppressed to prevent excessive email. For a list of all the issues modified, search jira comments for this (hopefully) unique string: 20080415hossman2

            People

            • Assignee:
              Erik Hatcher
              Reporter:
              Thorsten Scherler
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development