Solr
  1. Solr
  2. SOLR-20

A simple Java client for updating and searching

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.3
    • Component/s: clients - java
    • Labels:
      None
    • Environment:

      all

      Description

      I wrote a simple little client class that can connect to a Solr server and issue add, delete, commit and optimize commands using Java methods. I'm posting here for review and comments as suggested by Yonik.

      1. DocumentManagerClient.java
        12 kB
        Darren Erik Vengroff
      2. DocumentManagerClient.java
        12 kB
        Darren Erik Vengroff
      3. solrclient_addqueryfacet.zip
        7 kB
        Chen Lei
      4. solr-client.zip
        136 kB
        Ryan McKinley
      5. solr-client.zip
        130 kB
        Ryan McKinley
      6. solr-client.zip
        554 kB
        Ryan McKinley
      7. SolrClientException.java
        0.5 kB
        Darren Erik Vengroff
      8. solr-client-java.zip
        6 kB
        Darren Erik Vengroff
      9. solr-client-java-2.zip.zip
        6 kB
        Darren Erik Vengroff
      10. solr-client-sources.jar
        13 kB
        Darren Erik Vengroff
      11. SolrServerException.java
        1 kB
        Darren Erik Vengroff

        Issue Links

          Activity

          Hide
          Darren Erik Vengroff added a comment -

          Previous version didn't properly escape the query in the deleteByQuery() case.

          Show
          Darren Erik Vengroff added a comment - Previous version didn't properly escape the query in the deleteByQuery() case.
          Hide
          Erik Hatcher added a comment -

          This looks quite good and well documented! Thanks for this contribution. The only issue my current project would have with this is the Map which prevents multiple fields of the same name from being added. I use a lot of multi-valued fields.

          Show
          Erik Hatcher added a comment - This looks quite good and well documented! Thanks for this contribution. The only issue my current project would have with this is the Map which prevents multiple fields of the same name from being added. I use a lot of multi-valued fields.
          Hide
          Erik Hatcher added a comment -

          One idea for dealing with multivalued fields is to check the type of the object in the Map and if is an array or Collection then iterate over it rather than just doing .toString() on it. Would that logic work for your use as well?

          Show
          Erik Hatcher added a comment - One idea for dealing with multivalued fields is to check the type of the object in the Map and if is an array or Collection then iterate over it rather than just doing .toString() on it. Would that logic work for your use as well?
          Hide
          Darren Erik Vengroff added a comment -

          Iterating over an array or Collection of values is a great suggestion. I'll change it and resubmit when I have some time later today.

          Show
          Darren Erik Vengroff added a comment - Iterating over an array or Collection of values is a great suggestion. I'll change it and resubmit when I have some time later today.
          Hide
          Darren Erik Vengroff added a comment -

          Here is the latest, incorporating Erik's suggestion about supporting multi-valued fields.

          BTW, is there any way to delete the older attached versions of this file from JIRA? There's no real need for them to be there any more.

          Show
          Darren Erik Vengroff added a comment - Here is the latest, incorporating Erik's suggestion about supporting multi-valued fields. BTW, is there any way to delete the older attached versions of this file from JIRA? There's no real need for them to be there any more.
          Hide
          Darren Erik Vengroff added a comment -

          New exception type for reporting server-side exceptions.

          Show
          Darren Erik Vengroff added a comment - New exception type for reporting server-side exceptions.
          Hide
          Darren Erik Vengroff added a comment -

          New client code that uses the new exception.

          Show
          Darren Erik Vengroff added a comment - New client code that uses the new exception.
          Hide
          Darren Erik Vengroff added a comment -

          Here's the latest. There is an abstract base class that handles client connection and request/response and two subclasses. One is as before, with java APIs, and the other is for cases where you have an XML document you want to transform and send to the server.

          Show
          Darren Erik Vengroff added a comment - Here's the latest. There is an abstract base class that handles client connection and request/response and two subclasses. One is as before, with java APIs, and the other is for cases where you have an XML document you want to transform and send to the server.
          Hide
          Yonik Seeley added a comment -

          Great! Now we need to figure out where it lives, and how to work out the dependencies (a solr-util.jar that a client could use, or perhaps just pull the needed class or two directly into the solr-client.jar)

          Show
          Yonik Seeley added a comment - Great! Now we need to figure out where it lives, and how to work out the dependencies (a solr-util.jar that a client could use, or perhaps just pull the needed class or two directly into the solr-client.jar)
          Hide
          Philip Jacob added a comment -

          delete() in the DocumentManagerClient ought to be doing this:

          <delete><id>1234</id></delete>

          It's currently doing this:

          <delete><query>1234</query></delete>

          Show
          Philip Jacob added a comment - delete() in the DocumentManagerClient ought to be doing this: <delete><id>1234</id></delete> It's currently doing this: <delete><query>1234</query></delete>
          Hide
          Darren Erik Vengroff added a comment -

          Good catch Philip. For the benefit of future downloaders, here's a complete zip file with everything including this fix.

          Show
          Darren Erik Vengroff added a comment - Good catch Philip. For the benefit of future downloaders, here's a complete zip file with everything including this fix.
          Hide
          Darren Erik Vengroff added a comment -

          Here is the latest version of the client code, in the form of solr-client-source.jar. The big difference here is that there are now two clients, DocumentManagerClient for adding, inserting, and updating, and SearchClient for searching. They share the same underlying communication mechanism, which consists of a low-level mechanism for doing queries and parsing responses by reading from an InputStream (see ResponseParser) and a slightly higher level mechanism that handles some of the XML for you (see XmlResponseParser).

          I've been building this as a seperate project with a Maven2 dependency on Solr, but if the source is dropped into the Solr source tree at the appropriate place I suspect it will just compile and work. There are no other outside dependencies.

          I have some unit tests as well, but for the moment they are too tied in to my environment to be useful to the broader community. I will correct this and submit them.

          Show
          Darren Erik Vengroff added a comment - Here is the latest version of the client code, in the form of solr-client-source.jar. The big difference here is that there are now two clients, DocumentManagerClient for adding, inserting, and updating, and SearchClient for searching. They share the same underlying communication mechanism, which consists of a low-level mechanism for doing queries and parsing responses by reading from an InputStream (see ResponseParser) and a slightly higher level mechanism that handles some of the XML for you (see XmlResponseParser). I've been building this as a seperate project with a Maven2 dependency on Solr, but if the source is dropped into the Solr source tree at the appropriate place I suspect it will just compile and work. There are no other outside dependencies. I have some unit tests as well, but for the moment they are too tied in to my environment to be useful to the broader community. I will correct this and submit them.
          Hide
          Darren Erik Vengroff added a comment -

          Please ignore that last attachment. It contains an earlier version of the code than I intended, and has a couple of serious bugs. I'm sure that what I'm running now is a lot better, but I'm going to iterate a little more and build some more complete tests before I submit it again. Unless anyone out there is dieing to be on the bleeding edge of this.

          Show
          Darren Erik Vengroff added a comment - Please ignore that last attachment. It contains an earlier version of the code than I intended, and has a couple of serious bugs. I'm sure that what I'm running now is a lot better, but I'm going to iterate a little more and build some more complete tests before I submit it again. Unless anyone out there is dieing to be on the bleeding edge of this.
          Hide
          Yonik Seeley added a comment -

          Ping...
          any chance you could make your latest version available?

          Show
          Yonik Seeley added a comment - Ping... any chance you could make your latest version available?
          Hide
          Fuad Efendi added a comment -

          I'm dieing... which files should I download?

          Show
          Fuad Efendi added a comment - I'm dieing... which files should I download?
          Hide
          David Halsted added a comment -

          it is a little confusing – any chance one of the attachments could be designated as the right one? Thanks!

          Show
          David Halsted added a comment - it is a little confusing – any chance one of the attachments could be designated as the right one? Thanks!
          Hide
          Hoss Man added a comment -

          I don't know that there is a specific "right" one at the moment ... Darren's last comment suggests that he has a better version but it's not quite ready for submission.

          FYI: if you click the "All" link at teh top left of the comment listing, you can see where in the flow of time each of the attachemnts was added – from there it's pretty easy to tell that while solr-client-sources.jar is the most recent attachment, it's the one Darren said should be ignored...

          at this moment solr-client-java-2.zip.zip seems to be the most recent "good" version.

          Show
          Hoss Man added a comment - I don't know that there is a specific "right" one at the moment ... Darren's last comment suggests that he has a better version but it's not quite ready for submission. FYI: if you click the "All" link at teh top left of the comment listing, you can see where in the flow of time each of the attachemnts was added – from there it's pretty easy to tell that while solr-client-sources.jar is the most recent attachment, it's the one Darren said should be ignored... at this moment solr-client-java-2.zip.zip seems to be the most recent "good" version.
          Hide
          Otis Gospodnetic added a comment -

          SOLR-20 and SOLR-30 seem to be lingering in JIRA.
          Is the plan to merge them, and get the code into Solr?

          Show
          Otis Gospodnetic added a comment - SOLR-20 and SOLR-30 seem to be lingering in JIRA. Is the plan to merge them, and get the code into Solr?
          Hide
          Fuad Efendi added a comment -

          SOLR-20: add, delete, commit, optimize
          SOLR-30: search

          So, should be merged.

          HttpClient seems to be right choice (easily configurable; 'follow redirects', 'buffer size', etc.).

          Possible improvements:

          • make it independent on implementation (for instance, use interface and HttpClient-based implementation)
          • XML-based external configuration file
          • no need for JDOM...

          Probably, we need separate src/client folder for source files (sources do not depend on SOLR)

          Show
          Fuad Efendi added a comment - SOLR-20 : add, delete, commit, optimize SOLR-30 : search So, should be merged. HttpClient seems to be right choice (easily configurable; 'follow redirects', 'buffer size', etc.). Possible improvements: make it independent on implementation (for instance, use interface and HttpClient-based implementation) XML-based external configuration file no need for JDOM... Probably, we need separate src/client folder for source files (sources do not depend on SOLR)
          Hide
          Ryan McKinley added a comment -

          I've attached the client code i have written that merges SOLR-20 & SOLR-30. It can add, delete, commit, optimize and search.

          The main interface is SolrClient.

          It uses the XPP parser to parse the search results.

          If there is interist, i can clean it up some more...

          Show
          Ryan McKinley added a comment - I've attached the client code i have written that merges SOLR-20 & SOLR-30 . It can add, delete, commit, optimize and search. The main interface is SolrClient. It uses the XPP parser to parse the search results. If there is interist, i can clean it up some more...
          Hide
          Fuad Efendi added a comment -

          My previous comment is visible to jira-users only, sorry.
          Code submitted by Ryan looks great!

          Show
          Fuad Efendi added a comment - My previous comment is visible to jira-users only, sorry. Code submitted by Ryan looks great!
          Hide
          Ryan McKinley added a comment -

          I just posted a new version of a java client. This moves things to proper org.apache... packages and adds the waitFlush, waitSearcher suggested by Fuad.

          If people are interested, i think this should sit next to /client/ruby in: /client/java/solrj/

          there is a build.xml file that will generate a solr-client.jar file.

          As a taste, this is how you perform a search:
          <code>
          SolrClient client = new SolrClientImpl( new URL("http://localhost:8983/solr/") );

          SolrQuery query = new SolrQuery();
          query.setQuery( "video" );

          QueryResults results = client.query( query );

          for( ResultDoc doc : results.getDocs() )

          { System.out.println( "["+doc.getId()+"] "+ doc.getField( "name" ) ); }

          </code>

          Also, if there is interest, i can post an example webapp using this client library to search and explore a solr repository.

          Show
          Ryan McKinley added a comment - I just posted a new version of a java client. This moves things to proper org.apache... packages and adds the waitFlush, waitSearcher suggested by Fuad. If people are interested, i think this should sit next to /client/ruby in: /client/java/solrj/ there is a build.xml file that will generate a solr-client.jar file. As a taste, this is how you perform a search: <code> SolrClient client = new SolrClientImpl( new URL("http://localhost:8983/solr/") ); SolrQuery query = new SolrQuery(); query.setQuery( "video" ); QueryResults results = client.query( query ); for( ResultDoc doc : results.getDocs() ) { System.out.println( "["+doc.getId()+"] "+ doc.getField( "name" ) ); } </code> Also, if there is interest, i can post an example webapp using this client library to search and explore a solr repository.
          Hide
          Ryan McKinley added a comment -

          added APL to zip

          Show
          Ryan McKinley added a comment - added APL to zip
          Hide
          Hoss Man added a comment -

          (NOTE: revised summary since this issue has moved beyond just updating)

          I finally had a chance to look this over, here's a few comments in
          no particular order...

          1) i like the name solrj, i think this code should definitely live in client/java/solrj so that there is the potential for other java client code that is independent (if nothing else, i suspect something like SOLR-86 might be handy) ... we should probably put solrj in the package name as well.

          2) i wouldn't worry about having a special package for the exceptions ... they've got exception in their name, no ones going to be confused.

          3) I'm really not fond of "ParamNames.java" being a copy of the constants in "SolrParams.java", or XML.java being copied, or the xpp jar being duplicated ... it seems like we should just pull in those (compiled) classes at build time ... but that would require that the whole Solr tree be checked out, and there seems to be interest in making it possible to "svn checkout client/lang/impl" and build that in isolation ... perhaps we could use svn:externals to pull in specific utility classes and jars from other places in the tree? (although based on what I've read today, branching for releases would be hard since all of the svn:external props would have to be updated).

          what do people think in general about how the client code can/should/shouldn't depend on the core server code?

          4) one thing we should really try to support in a client is executing query requests against non-standard request handlers ... handlers that might take in request params that we can't even imagine. The SolrQuery class has explicit setters for many of the params that the built in request handlers support, but there is no easy way for people to build other queries. I think it might make sense if SolrQuery was an interface that just defined the methods needed by the SolrClient – probably just getQueryString(). Then their can be a SimpleSolrQuery that has all of the setters in the current SolrQuery class, possibly using a general baseclass with an impl of getQueryString that uses some SolrParams...

          public class AbstractSolrQuery implements SolrQuery {
          protected abstract SolrParams getSolrParams();
          public String getQueryString()

          { ... your current code, looping over getSolrParams() ... }

          }

          5) what is the purpose of SolrClientStub ?

          6) what is the purpose of SolrDocumentable being an empty interface? ... it seems like you could replace SolrDocumentable, SolrDocument, and SolrDocumented with something like this...

          public interface SolrDocument

          { public Map<String,Object> getSolrDocumentFields(); }

          public abstract class SolrDocumented implements SolrDocument {
          protected abstract SolrDocument getSolrDocument();
          public Map<String,Object> getSolrDocumentFields()

          { return getSolrDocument().getSolrDocumentFields() }

          }

          Then you wouldn't need that instanceof code in SolrClientImpl

          Note that we should probably support field and document boosts as well, but field boosts don't really need to be specified in the Map since they apply to the whole field and not the individual values, so we could just add...

          public int getDocumentBoost();
          public Map<String,Integer> getFieldBoosts()

          ...to SolrDocument.

          7) The ResultsParser and QueryResults classes seem to suffer the same limitation that i was mentioning about the SolrQuery class – they assume a very specific response structure (only one doc list, an optional facet block, an optional highlighting block, an optional debug block) ... I think since the ResultsParser already understands the all of the various tags that are used, it should be easy to do this as long as the QueryResult object becomes a more general container that any named data can be shoved into (just like SolrQueryResponse is on the server side) ... then a "SimpleQueryResults" class could be written that had the convenience methods that make sense when using StandardRequestHandler or DisMaxRequestHandler.

          8) There was a comment in SOLR-30 regarding the issue of that code only parsing the XML response ... i think it's completely practical to focus on client code which currently supports only the XmlResponseWriter output – especially with the solrj ResultsParser class currently having a single public method...

          public QueryResults process( Reader reader ) throws SolrClientException, SolrServerException, XmlPullParserException, IOException

          ...i think if we removed XmlPullParserException from that list of exceptions (it could always be wrapped in a SolrClientException, or a new SolrClientParseException) we have a really simple API where other ResultParser classes could be written to handle JSON or what not down the road just by adding a simple setResultParser to SolrClient.

          Show
          Hoss Man added a comment - (NOTE: revised summary since this issue has moved beyond just updating) I finally had a chance to look this over, here's a few comments in no particular order... 1) i like the name solrj, i think this code should definitely live in client/java/solrj so that there is the potential for other java client code that is independent (if nothing else, i suspect something like SOLR-86 might be handy) ... we should probably put solrj in the package name as well. 2) i wouldn't worry about having a special package for the exceptions ... they've got exception in their name, no ones going to be confused. 3) I'm really not fond of "ParamNames.java" being a copy of the constants in "SolrParams.java", or XML.java being copied, or the xpp jar being duplicated ... it seems like we should just pull in those (compiled) classes at build time ... but that would require that the whole Solr tree be checked out, and there seems to be interest in making it possible to "svn checkout client/lang/impl" and build that in isolation ... perhaps we could use svn:externals to pull in specific utility classes and jars from other places in the tree? (although based on what I've read today, branching for releases would be hard since all of the svn:external props would have to be updated). what do people think in general about how the client code can/should/shouldn't depend on the core server code? 4) one thing we should really try to support in a client is executing query requests against non-standard request handlers ... handlers that might take in request params that we can't even imagine. The SolrQuery class has explicit setters for many of the params that the built in request handlers support, but there is no easy way for people to build other queries. I think it might make sense if SolrQuery was an interface that just defined the methods needed by the SolrClient – probably just getQueryString(). Then their can be a SimpleSolrQuery that has all of the setters in the current SolrQuery class, possibly using a general baseclass with an impl of getQueryString that uses some SolrParams... public class AbstractSolrQuery implements SolrQuery { protected abstract SolrParams getSolrParams(); public String getQueryString() { ... your current code, looping over getSolrParams() ... } } 5) what is the purpose of SolrClientStub ? 6) what is the purpose of SolrDocumentable being an empty interface? ... it seems like you could replace SolrDocumentable, SolrDocument, and SolrDocumented with something like this... public interface SolrDocument { public Map<String,Object> getSolrDocumentFields(); } public abstract class SolrDocumented implements SolrDocument { protected abstract SolrDocument getSolrDocument(); public Map<String,Object> getSolrDocumentFields() { return getSolrDocument().getSolrDocumentFields() } } Then you wouldn't need that instanceof code in SolrClientImpl Note that we should probably support field and document boosts as well, but field boosts don't really need to be specified in the Map since they apply to the whole field and not the individual values, so we could just add... public int getDocumentBoost(); public Map<String,Integer> getFieldBoosts() ...to SolrDocument. 7) The ResultsParser and QueryResults classes seem to suffer the same limitation that i was mentioning about the SolrQuery class – they assume a very specific response structure (only one doc list, an optional facet block, an optional highlighting block, an optional debug block) ... I think since the ResultsParser already understands the all of the various tags that are used, it should be easy to do this as long as the QueryResult object becomes a more general container that any named data can be shoved into (just like SolrQueryResponse is on the server side) ... then a "SimpleQueryResults" class could be written that had the convenience methods that make sense when using StandardRequestHandler or DisMaxRequestHandler. 8) There was a comment in SOLR-30 regarding the issue of that code only parsing the XML response ... i think it's completely practical to focus on client code which currently supports only the XmlResponseWriter output – especially with the solrj ResultsParser class currently having a single public method... public QueryResults process( Reader reader ) throws SolrClientException, SolrServerException, XmlPullParserException, IOException ...i think if we removed XmlPullParserException from that list of exceptions (it could always be wrapped in a SolrClientException, or a new SolrClientParseException) we have a really simple API where other ResultParser classes could be written to handle JSON or what not down the road just by adding a simple setResultParser to SolrClient.
          Hide
          J.J. Larrea added a comment -

          Regarding Hoss' point #3, perhaps it's time to reorganize into something like

          /solr/server/...
          /solr/client/...
          /solr/webapp/,,, (or /solr/server/webapp)
          /solr/shared/...

          "To build client XXX check out /solr/client or just /solr/client/java/XXX and /solr/shared"

          Shared would include external constants and exceptions.

          Show
          J.J. Larrea added a comment - Regarding Hoss' point #3, perhaps it's time to reorganize into something like /solr/server/... /solr/client/... /solr/webapp/,,, (or /solr/server/webapp) /solr/shared/... "To build client XXX check out /solr/client or just /solr/client/java/XXX and /solr/shared" Shared would include external constants and exceptions.
          Hide
          Ryan McKinley added a comment -

          I have dramatically reworked the client code to fit with the pluggable ContentStream model in SOLR-104. This version makes it easy to customize/extend the request/response behavior. Once it stabilizes and is better tested, I'll upload a zip, but for now, you can preview it:

          http://svn.lapnap.net/solr/solrj/

          Major changes:

          • it is based on commons-httpclient-3.0.1.jar
          • I'm using wt=JSON rather then XML. (It maps to a hash easier)
          • I moved some of the common classes to o.a.s.util. Hopefully the core classes will be refactored to make it easier to share some classes
          • handles multiple ContentStreams using multi-part form upload
          • Got rid of the SolrDocumentable/SolrDocumented distinction. – now there is only SolrDocument()
          • You can define and automatically build a solr document with annotations
          • Includes a first draft for a HibernateEventListener. When stuff is added/updated/deleted, it gets sent to solr. (Note, this class should probable not be in the main client as the hibernate prerequisite libraries are substantial - I've included them because its what i need to have working soon) When this is more stable, it will be something similar to a Compass Hibernate3GpsDevice (http://www.opensymphony.com/compass/versions/1.1RC1/html/gps-hibernate.html)

          The key interfaces are:

          public interface SolrClient

          { public abstract SolrResponse process( final SolrRequest req ); }

          public interface SolrRequest

          { public String getMethod(); public String getHandlerPath(); public RequestParams getParams(); public Collection<ContentStream> getContentStreams(); public SolrResponse parseResponseBody(InputStream in); public SolrResponse execute(SolrClient solr); }
          • - - - - - -
            Here is some sample usage:

          SolrClient client = new CommonsHttpSolrClient(
          new URL("http://localhost:8983/solr/") );

          // Set up a simple query
          SolrQuery query = new SolrQuery();
          query.setQuery( "solr" );
          query.addFacetField( "cat" );
          query.setFacetLimit( 15 );
          query.setQuery( "video" );
          query.setShowDebugInfo( true );

          QueryResponse rsp = query.execute( client );
          for( ResultDoc doc : rsp.getDocs() )

          { System.out.println( doc.get( "name" ) ); System.out.println( doc.getScore() ); System.out.println( doc.getExplain() ); }

          SimpleSolrDoc doc = new SimpleSolrDoc();
          doc.setField( "id", "xxx" );
          doc.setField( "price", 12.34f );
          doc.setField( "cat", new String[]

          { "aaa", "bbb", "ccc" }

          );
          new AddDocuments( doc ).execute( client );
          new CommitIndex().execute( client );

          • - - - - - - - - - -

          This also includes a utility to make solr documents from annotations. Given the class:

          @SolrSearchable( boost=2.0 )
          public class Example
          {
          @SolrSearchable
          public String getName()

          { return "hello" }

          @SolrSearchable( name="cat", boost=3 )
          public String getSomeOtherName()

          { return "there" }

          }

          The DocumentBuilder can automatically make:

          <doc boost="2.0">
          <field name="name">hello</field>
          <field name="cat" boost="3">there</field>
          </doc>

          • - - - - - - - - - -

          There are a few parts of the API i think are awkward, I'd love any feedback / review you may have.

          thanks
          ryan

          Show
          Ryan McKinley added a comment - I have dramatically reworked the client code to fit with the pluggable ContentStream model in SOLR-104 . This version makes it easy to customize/extend the request/response behavior. Once it stabilizes and is better tested, I'll upload a zip, but for now, you can preview it: http://svn.lapnap.net/solr/solrj/ Major changes: it is based on commons-httpclient-3.0.1.jar I'm using wt=JSON rather then XML. (It maps to a hash easier) I moved some of the common classes to o.a.s.util. Hopefully the core classes will be refactored to make it easier to share some classes handles multiple ContentStreams using multi-part form upload Got rid of the SolrDocumentable/SolrDocumented distinction. – now there is only SolrDocument() You can define and automatically build a solr document with annotations Includes a first draft for a HibernateEventListener. When stuff is added/updated/deleted, it gets sent to solr. (Note, this class should probable not be in the main client as the hibernate prerequisite libraries are substantial - I've included them because its what i need to have working soon) When this is more stable, it will be something similar to a Compass Hibernate3GpsDevice ( http://www.opensymphony.com/compass/versions/1.1RC1/html/gps-hibernate.html ) The key interfaces are: public interface SolrClient { public abstract SolrResponse process( final SolrRequest req ); } public interface SolrRequest { public String getMethod(); public String getHandlerPath(); public RequestParams getParams(); public Collection<ContentStream> getContentStreams(); public SolrResponse parseResponseBody(InputStream in); public SolrResponse execute(SolrClient solr); } - - - - - - Here is some sample usage: SolrClient client = new CommonsHttpSolrClient( new URL("http://localhost:8983/solr/") ); // Set up a simple query SolrQuery query = new SolrQuery(); query.setQuery( "solr" ); query.addFacetField( "cat" ); query.setFacetLimit( 15 ); query.setQuery( "video" ); query.setShowDebugInfo( true ); QueryResponse rsp = query.execute( client ); for( ResultDoc doc : rsp.getDocs() ) { System.out.println( doc.get( "name" ) ); System.out.println( doc.getScore() ); System.out.println( doc.getExplain() ); } SimpleSolrDoc doc = new SimpleSolrDoc(); doc.setField( "id", "xxx" ); doc.setField( "price", 12.34f ); doc.setField( "cat", new String[] { "aaa", "bbb", "ccc" } ); new AddDocuments( doc ).execute( client ); new CommitIndex().execute( client ); - - - - - - - - - - This also includes a utility to make solr documents from annotations. Given the class: @SolrSearchable( boost=2.0 ) public class Example { @SolrSearchable public String getName() { return "hello" } @SolrSearchable( name="cat", boost=3 ) public String getSomeOtherName() { return "there" } } The DocumentBuilder can automatically make: <doc boost="2.0"> <field name="name">hello</field> <field name="cat" boost="3">there</field> </doc> - - - - - - - - - - There are a few parts of the API i think are awkward, I'd love any feedback / review you may have. thanks ryan
          Hide
          Yonik Seeley added a comment -

          > * it is based on commons-httpclient-3.0.1.jar
          Cool, +1

          > * I'm using wt=JSON rather then XML. (It maps to a hash easier)
          Heh... I quickly checked out the code, but didn't see where you were parsing the code, or where the JSONObject class referenced is.

          Anyway, if you want the best JSON parser on the planet, check out
          http://www.nabble.com/Apache-Lab-proposal%3A-noggit-tf2701405.html#a7532843
          http://svn.apache.org/repos/asf/labs/noggit/

          I haven't had a chance to do the writing side, or the "create full object graph" part, but the parser is screaming fast.

          > * handles multiple ContentStreams using multi-part form upload
          Will a client need to do that? I had thought a browser would be the only one using multi-part

          > * You can define and automatically build a solr document with annotations
          Sounds cool

          > * Includes a first draft for a HibernateEventListener.
          Sounds very cool... it should go in a separate contrib eventually.

          Show
          Yonik Seeley added a comment - > * it is based on commons-httpclient-3.0.1.jar Cool, +1 > * I'm using wt=JSON rather then XML. (It maps to a hash easier) Heh... I quickly checked out the code, but didn't see where you were parsing the code, or where the JSONObject class referenced is. Anyway, if you want the best JSON parser on the planet, check out http://www.nabble.com/Apache-Lab-proposal%3A-noggit-tf2701405.html#a7532843 http://svn.apache.org/repos/asf/labs/noggit/ I haven't had a chance to do the writing side, or the "create full object graph" part, but the parser is screaming fast. > * handles multiple ContentStreams using multi-part form upload Will a client need to do that? I had thought a browser would be the only one using multi-part > * You can define and automatically build a solr document with annotations Sounds cool > * Includes a first draft for a HibernateEventListener. Sounds very cool... it should go in a separate contrib eventually.
          Hide
          rubdabadub added a comment -

          Hi:

          I was really hoping that this patch will make it to trunk soon. I been using it without any problem. I was wondering if there are any specifics that are left for SOLR-20 to make it to trunk? I would be more then happy to help out anyway I can. I need SOLR-20 for Nutch-Solr integration so it would be nice if it was included.

          Thankful for your kind attention to SOLR-20.

          Show
          rubdabadub added a comment - Hi: I was really hoping that this patch will make it to trunk soon. I been using it without any problem. I was wondering if there are any specifics that are left for SOLR-20 to make it to trunk? I would be more then happy to help out anyway I can. I need SOLR-20 for Nutch-Solr integration so it would be nice if it was included. Thankful for your kind attention to SOLR-20 .
          Hide
          Erik Hatcher added a comment -

          I want this on trunk also. I'll be reviewing it and testing it out this afternoon and committing if all is fine.

          Show
          Erik Hatcher added a comment - I want this on trunk also. I'll be reviewing it and testing it out this afternoon and committing if all is fine.
          Hide
          Erik Hatcher added a comment -

          My bad... it was SOLR-86 that interested me, and I've just committed it. I can't take on these patches just yet.

          Show
          Erik Hatcher added a comment - My bad... it was SOLR-86 that interested me, and I've just committed it. I can't take on these patches just yet.
          Hide
          Ryan McKinley added a comment -

          I don't think this one is quite ready to go, but anyone interested can see an updated versions at:

          http://solrstuff.org/svn/solrj/
          http://solrstuff.org/svn/solrj-hibernate/

          I have extracted the hibernate specific stuff into its own project so solrj is a bit more manageable.

          • - - - -

          If you are looking for a stable, simple client "solr-client.zip" is still your best bet.

          Show
          Ryan McKinley added a comment - I don't think this one is quite ready to go, but anyone interested can see an updated versions at: http://solrstuff.org/svn/solrj/ http://solrstuff.org/svn/solrj-hibernate/ I have extracted the hibernate specific stuff into its own project so solrj is a bit more manageable. - - - - If you are looking for a stable, simple client "solr-client.zip" is still your best bet.
          Hide
          Ryan McKinley added a comment -

          to do SOLR-20 properly, it should use shared libraries rather then redefine things like constants and common utility classes.

          Show
          Ryan McKinley added a comment - to do SOLR-20 properly, it should use shared libraries rather then redefine things like constants and common utility classes.
          Hide
          Frederic Hennequin added a comment -

          Hello, we have been testing the solr-client and think we have found a small bug :

          the xml parsers on the query-side is not setup to use "UTF-8" encoding
          this resulted in weird characters being returned by the solr-client...
          for example : "é" came out like "©" ... not really what we would like...

          we fixed it by setting the input stream for the xmlparser to "UTF8" which gave us this code in ResultsParser.java :
          [code]
          public QueryResults process( InputStream reader ) throws SolrClientException, SolrServerException, XmlPullParserException, IOException
          {
          QueryResults res = new QueryResults();

          try {
          XmlPullParser xpp = null;
          try

          { xpp = factory.newPullParser(); xpp.setInput(reader,"UTF-8"); xpp.nextTag(); }


          .....
          [/code]

          notice we changed the argument for this method to InputStream instead of the reader so we could add "UTF-8" to the stream.
          by doing this we had to change the reader in SolrClientImpl.java to an inputstream :
          [code]
          ....
          InputStream inputStream = urlc.getInputStream();
          try

          { QueryResults res = parser.process( inputStream ); res.setSolrURL( qurl ); res.setQuery( query ); return res; }

          ....
          [/code]

          in our opinion this was a major bug (since all solr-xml is encoded in utf-8) and we guess somebody just forgot to put it in...

          yay, now we can all start using freaky characters without the client actually freaking out enjoy

          Show
          Frederic Hennequin added a comment - Hello, we have been testing the solr-client and think we have found a small bug : the xml parsers on the query-side is not setup to use "UTF-8" encoding this resulted in weird characters being returned by the solr-client... for example : "é" came out like "©" ... not really what we would like... we fixed it by setting the input stream for the xmlparser to "UTF8" which gave us this code in ResultsParser.java : [code] public QueryResults process( InputStream reader ) throws SolrClientException, SolrServerException, XmlPullParserException, IOException { QueryResults res = new QueryResults(); try { XmlPullParser xpp = null; try { xpp = factory.newPullParser(); xpp.setInput(reader,"UTF-8"); xpp.nextTag(); } ..... [/code] notice we changed the argument for this method to InputStream instead of the reader so we could add "UTF-8" to the stream. by doing this we had to change the reader in SolrClientImpl.java to an inputstream : [code] .... InputStream inputStream = urlc.getInputStream(); try { QueryResults res = parser.process( inputStream ); res.setSolrURL( qurl ); res.setQuery( query ); return res; } .... [/code] in our opinion this was a major bug (since all solr-xml is encoded in utf-8) and we guess somebody just forgot to put it in... yay, now we can all start using freaky characters without the client actually freaking out enjoy
          Hide
          Thierry Collogne added a comment -

          I found that there was no way of adding highlight paramters to an SolrQuery, so I made some modifications to SolrQuery.java and ParamNames.java to allow highlighting

          Changes to SolrQuery

          First add a new variable

          private HighlightParams _highlight = new HighlightParams();

          Than I defined a new innerclass (a bit like FacetParams)

          public static class HighlightParams {
          public List<String> field = new ArrayList<String>();
          public int snippets = 3;
          public int fragsize = 100;

          public String simple_pre = "<b>";
          public String simple_post = "</b>";

          public boolean isEnabled()

          { return field.size() > 0; }

          }

          Than add the following methods

          public void addHighlightField( String f )

          { _highlight.field.add( f ); }

          public void setHighlightSnippets( int snippets )

          { _highlight.snippets = snippets; }

          public void setHighlightFragSize( int fragsize )

          { _highlight.fragsize = fragsize; }

          // ATTENTION : only simple tags. No quotes like in <span class="tag">
          public void setHighlightSurroundingTags( String pre, String post )

          { _highlight.simple_pre = pre; _highlight.simple_post = post; }

          Add the following code to getQueryString method (for example right before return builder.toString()

          if( _highlight.isEnabled() ) {
          builder.append( '&' ).append( ParamNames.HIGHLIGHT ).append( "=true" );
          builder.append( '&' ).append( ParamNames.HIGHLIGHT_FIELDS ).append( "=" );
          for( String f : _highlight.field )

          { builder.append( f ).append( ',' ); }

          builder.append( '&' ).append( ParamNames.HIGHLIGHT_SNIPPETS ).append( "=" ).append( _highlight.snippets );
          builder.append( '&' ).append( ParamNames.HIGHLIGHT_FRAGSIZE ).append( "=" ).append( _highlight.fragsize );

          builder.append( '&' ).append( ParamNames.HIGHLIGHT_SIMPLE_PRE ).append( "=" ).append( _highlight.simple_pre );
          builder.append( '&' ).append( ParamNames.HIGHLIGHT_SIMPLE_POST ).append( "=" ).append( _highlight.simple_post );
          }

          Changes to ParamNames.java

          Add/modify the following variables

          /** wether to highlight */
          public static final String HIGHLIGHT = "hl";
          /** fields to highlight */
          public static final String HIGHLIGHT_FIELDS = "hl.fl";
          /** maximum highlight fragments to return */
          public static final String HIGHLIGHT_SNIPPETS = "hl.snippets";
          /** override default highlight fragsize */
          public static final String HIGHLIGHT_FRAGSIZE = "hl.fragsize";
          /** override default pre highlight */
          public static final String HIGHLIGHT_SIMPLE_PRE = "hl.simple.pre";
          /** override default post highlight */
          public static final String HIGHLIGHT_SIMPLE_POST = "hl.simple.post";

          This can be used as follwed
          SolrClient client = new SolrClientImpl( new URL("http://localhost:8080/solr/") );
          // required
          query.addHighlightField("title");
          query.addHighlightField("content");
          // following is optional
          query.setHighlightFragSize(100);
          query.setHighlightSnippets(3);
          query.setHighlightSurroundingTags("<i>","</i>");

          I think that is all. If I forgot something, post it here. One remark. The setHighlightSurroundingTags method can only take simple tags,
          no tags containing quotes or such.

          Greetz.

          Show
          Thierry Collogne added a comment - I found that there was no way of adding highlight paramters to an SolrQuery, so I made some modifications to SolrQuery.java and ParamNames.java to allow highlighting Changes to SolrQuery First add a new variable private HighlightParams _highlight = new HighlightParams(); Than I defined a new innerclass (a bit like FacetParams) public static class HighlightParams { public List<String> field = new ArrayList<String>(); public int snippets = 3; public int fragsize = 100; public String simple_pre = "<b>"; public String simple_post = "</b>"; public boolean isEnabled() { return field.size() > 0; } } Than add the following methods public void addHighlightField( String f ) { _highlight.field.add( f ); } public void setHighlightSnippets( int snippets ) { _highlight.snippets = snippets; } public void setHighlightFragSize( int fragsize ) { _highlight.fragsize = fragsize; } // ATTENTION : only simple tags. No quotes like in <span class="tag"> public void setHighlightSurroundingTags( String pre, String post ) { _highlight.simple_pre = pre; _highlight.simple_post = post; } Add the following code to getQueryString method (for example right before return builder.toString() if( _highlight.isEnabled() ) { builder.append( '&' ).append( ParamNames.HIGHLIGHT ).append( "=true" ); builder.append( '&' ).append( ParamNames.HIGHLIGHT_FIELDS ).append( "=" ); for( String f : _highlight.field ) { builder.append( f ).append( ',' ); } builder.append( '&' ).append( ParamNames.HIGHLIGHT_SNIPPETS ).append( "=" ).append( _highlight.snippets ); builder.append( '&' ).append( ParamNames.HIGHLIGHT_FRAGSIZE ).append( "=" ).append( _highlight.fragsize ); builder.append( '&' ).append( ParamNames.HIGHLIGHT_SIMPLE_PRE ).append( "=" ).append( _highlight.simple_pre ); builder.append( '&' ).append( ParamNames.HIGHLIGHT_SIMPLE_POST ).append( "=" ).append( _highlight.simple_post ); } Changes to ParamNames.java Add/modify the following variables /** wether to highlight */ public static final String HIGHLIGHT = "hl"; /** fields to highlight */ public static final String HIGHLIGHT_FIELDS = "hl.fl"; /** maximum highlight fragments to return */ public static final String HIGHLIGHT_SNIPPETS = "hl.snippets"; /** override default highlight fragsize */ public static final String HIGHLIGHT_FRAGSIZE = "hl.fragsize"; /** override default pre highlight */ public static final String HIGHLIGHT_SIMPLE_PRE = "hl.simple.pre"; /** override default post highlight */ public static final String HIGHLIGHT_SIMPLE_POST = "hl.simple.post"; This can be used as follwed SolrClient client = new SolrClientImpl( new URL("http://localhost:8080/solr/") ); // required query.addHighlightField("title"); query.addHighlightField("content"); // following is optional query.setHighlightFragSize(100); query.setHighlightSnippets(3); query.setHighlightSurroundingTags("<i>","</i>"); I think that is all. If I forgot something, post it here. One remark. The setHighlightSurroundingTags method can only take simple tags, no tags containing quotes or such. Greetz.
          Hide
          Chen Lei added a comment -

          org.apache.solr.client.impl.ResultsParser.java
          org.apache.solr.client.QueryFacet.java (a copy of FieldFacet)
          org.apache.solr.client.QueryResults.java
          org.apache.solr.client.SolrQuery.java

          I changed these four files, add some functions for facet query.
          Add a facet query is similar as field query.
          Existing methods, new methods:
          SolrQuery.addFacetField() SolrQuery.addFacetQuery
          QueryResults.getFacets() QueryResults.getQueryFacets()
          QueryResults.getLimitingFacets QueryResults.getLimitingFacetsForQueryFacet()

          Some code here..
          SolrQuery query = new SolrQuery();
          // ...
          query.addFacetQuery("cat:music card");
          query.addFacetQuery("video");

          SolrClient client = ..
          // ...
          QueryResults results = client.query( query );

          for (QueryFacet qf : results.getQueryFacets()) {
          System.out.println("query facet: "+qf.getName() +" "+qf.getValueCount());
          for (org.apache.solr.client.QueryFacet.Count c : qf.getValues() )

          { System.out.println(" "+c.getName()+": "+c.getCount()); }

          }

          Show
          Chen Lei added a comment - org.apache.solr.client.impl.ResultsParser.java org.apache.solr.client.QueryFacet.java (a copy of FieldFacet) org.apache.solr.client.QueryResults.java org.apache.solr.client.SolrQuery.java I changed these four files, add some functions for facet query. Add a facet query is similar as field query. Existing methods, new methods: SolrQuery.addFacetField() SolrQuery.addFacetQuery QueryResults.getFacets() QueryResults.getQueryFacets() QueryResults.getLimitingFacets QueryResults.getLimitingFacetsForQueryFacet() Some code here.. SolrQuery query = new SolrQuery(); // ... query.addFacetQuery("cat:music card"); query.addFacetQuery("video"); SolrClient client = .. // ... QueryResults results = client.query( query ); for (QueryFacet qf : results.getQueryFacets()) { System.out.println("query facet: "+qf.getName() +" "+qf.getValueCount()); for (org.apache.solr.client.QueryFacet.Count c : qf.getValues() ) { System.out.println(" "+c.getName()+": "+c.getCount()); } }
          Hide
          carlos orrego added a comment -

          I am using the client library nad have no issues with adding docs to solr. But when i want to make a simple test query i keep getting this error:

          org.apache.solr.client.exception.SolrClientException: unknown type: status

          Solr returns and xml with a statis tag in the header wich the client does not know how to handle?

          am i right? is this the case?

          Show
          carlos orrego added a comment - I am using the client library nad have no issues with adding docs to solr. But when i want to make a simple test query i keep getting this error: org.apache.solr.client.exception.SolrClientException: unknown type: status Solr returns and xml with a statis tag in the header wich the client does not know how to handle? am i right? is this the case?
          Hide
          Will Johnson added a comment -

          is there an updated package or anyone working on such a thing at the moment? the solr-client.zip at the top of the thread works like a charm but seems to be very outdated and the bits on the svn://solrstuff.org site have some rather serious bugs.

          i'm happy to do all the leg work of packaging things, fixing bugs, submitting a patch, etc but i wanted to make sure i'm not about to walk right behind someone else. also, if anyone has any ideas for the best starting point i'm happy to take suggestions.

          • will
          Show
          Will Johnson added a comment - is there an updated package or anyone working on such a thing at the moment? the solr-client.zip at the top of the thread works like a charm but seems to be very outdated and the bits on the svn://solrstuff.org site have some rather serious bugs. i'm happy to do all the leg work of packaging things, fixing bugs, submitting a patch, etc but i wanted to make sure i'm not about to walk right behind someone else. also, if anyone has any ideas for the best starting point i'm happy to take suggestions. will
          Hide
          Ryan McKinley added a comment -

          For now, I'd still recommend using solr-client.zip

          The code on solrstuff.org is in the middle of a big overhaul (hopefully stable by the end of this week) – but it will rely on some changes to solr that are not likely to make it into solr-1.2.

          I'll post another message here when that settles down.

          thanks
          ryan

          Show
          Ryan McKinley added a comment - For now, I'd still recommend using solr-client.zip The code on solrstuff.org is in the middle of a big overhaul (hopefully stable by the end of this week) – but it will rely on some changes to solr that are not likely to make it into solr-1.2. I'll post another message here when that settles down. thanks ryan
          Hide
          Ryan McKinley added a comment -

          For anyone interested, I've finished a major overhaul of the client at:
          http://solrstuff.org/svn/solrj/
          It is a dramatically different architecture then before. Essentially it reads each response into a NamedList and each response type knows what the contents mean.

          After solr1.2, I'll work on getting something like this into the official apache distribution.

          This client source duplicates many classes that will eventually be extracted into an independent solr-utils.jar (SOLR-135). Everything that is not in "client"
          http://solrstuff.org/svn/solrj/src/org/apache/solr/ will be in this .jar

          I am using this in production code – but i don't suggest that it is production ready just yet.

          Show
          Ryan McKinley added a comment - For anyone interested, I've finished a major overhaul of the client at: http://solrstuff.org/svn/solrj/ It is a dramatically different architecture then before. Essentially it reads each response into a NamedList and each response type knows what the contents mean. After solr1.2, I'll work on getting something like this into the official apache distribution. This client source duplicates many classes that will eventually be extracted into an independent solr-utils.jar ( SOLR-135 ). Everything that is not in "client" http://solrstuff.org/svn/solrj/src/org/apache/solr/ will be in this .jar I am using this in production code – but i don't suggest that it is production ready just yet.
          Hide
          Will Johnson added a comment -

          the trunk version at http://solrstuff.org/svn/solrj/ seems to be missing a dependency and a copy of SolrParams. ant returns....

          compile:
          [javac] Compiling 40 source files to C:\data\workspace\solrj\bin
          [javac] C:\data\workspace\solrj\src\org\apache\solr\client\solrj\impl\XMLResponseParser.java:10: package javax.xml.stream does not exist
          [javac] import javax.xml.stream.XMLInputFactory;

          ....

          [javac] C:\data\workspace\solrj\src\org\apache\solr\client\solrj\query\SolrQuery.java:10: cannot find symbol
          [javac] symbol : class SolrParams
          [javac] location: package org.apache.solr.util
          [javac] import org.apache.solr.util.SolrParams;

          Show
          Will Johnson added a comment - the trunk version at http://solrstuff.org/svn/solrj/ seems to be missing a dependency and a copy of SolrParams. ant returns.... compile: [javac] Compiling 40 source files to C:\data\workspace\solrj\bin [javac] C:\data\workspace\solrj\src\org\apache\solr\client\solrj\impl\XMLResponseParser.java:10: package javax.xml.stream does not exist [javac] import javax.xml.stream.XMLInputFactory; .... [javac] C:\data\workspace\solrj\src\org\apache\solr\client\solrj\query\SolrQuery.java:10: cannot find symbol [javac] symbol : class SolrParams [javac] location: package org.apache.solr.util [javac] import org.apache.solr.util.SolrParams;
          Hide
          Ryan McKinley added a comment -

          aaah, It was compiling with java 6.

          I just added stax-api-1.0.jar and cleaned up some imports, it should run on java 5 now.

          Show
          Ryan McKinley added a comment - aaah, It was compiling with java 6. I just added stax-api-1.0.jar and cleaned up some imports, it should run on java 5 now.
          Hide
          Will Johnson added a comment -

          the new api's work great, thanks! what's the plan for this going forward? id' like to start doing some work on this as it's rather critical to my current project and an are i've dealt with a lot in the past. assuming it's not getting dumped into org.apache.* land any time soon are you accepting patches to this code? if so i have some modifications to the api's that i think will make them easier to use (such as a method to set FacetParams on SolrQuery) and i'll even flush out the SolrServerTest for fun.

          also, i noticed that all the methods on SolrServer throw undeclared SolrExceptions which extends RuntimeException when things so south. should those throw some other sort of non-ignorable exception like a new SolrServerException? while it made coding/compiling easier to leave out all the usually required try's and catches it made running/debugging much less enjoyable.

          • will
          Show
          Will Johnson added a comment - the new api's work great, thanks! what's the plan for this going forward? id' like to start doing some work on this as it's rather critical to my current project and an are i've dealt with a lot in the past. assuming it's not getting dumped into org.apache.* land any time soon are you accepting patches to this code? if so i have some modifications to the api's that i think will make them easier to use (such as a method to set FacetParams on SolrQuery) and i'll even flush out the SolrServerTest for fun. also, i noticed that all the methods on SolrServer throw undeclared SolrExceptions which extends RuntimeException when things so south. should those throw some other sort of non-ignorable exception like a new SolrServerException? while it made coding/compiling easier to leave out all the usually required try's and catches it made running/debugging much less enjoyable. will
          Hide
          Ryan McKinley added a comment -

          great! Any feedback/help would be wonderful.

          I hope it is not too long before this can enter solr trunk, but it will first need two solr1.3 additions SOLR-193 and SOLR-135, until then I can apply any patches necessary.

          Re RuntimeException vs SolrServerException, I'm not sure the best choice. Earlier versions had a client exception and server exception, but in practice those got lumped together (in my case) anyway. I ended up just using SolrException because it is there.

          Show
          Ryan McKinley added a comment - great! Any feedback/help would be wonderful. I hope it is not too long before this can enter solr trunk, but it will first need two solr1.3 additions SOLR-193 and SOLR-135 , until then I can apply any patches necessary. Re RuntimeException vs SolrServerException, I'm not sure the best choice. Earlier versions had a client exception and server exception, but in practice those got lumped together (in my case) anyway. I ended up just using SolrException because it is there.
          Hide
          Ben Incani added a comment -

          The SolrClientImpl does not implement the following optional attributes for "add" as documented in http://wiki.apache.org/solr/UpdateXmlMessages

          allowDups = "true" | "false" — default is "false"
          overwritePending = "true" | "false" — default is negation of allowDups
          overwriteCommitted = "true"|"false" — default is negation of allowDups

          Attached is patch for SolrClientImpl.java which implements allowDups.

              • SolrClientImpl.java.patch ***
                48a49,55
                >
                > /**
                > * Optional attributes for "add"
                > */
                > protected boolean allowDups;
                > protected boolean overwritePending; // TODO: not implemented
                > protected boolean overwriteCommitted; // TODO: not implemented
                86a94,97
                > public SolrClientImpl(URL baseURL) throws Exception { > this(baseURL, false); > }

                >
                91a103
                > * @param allowDups allow duplicates to be added to the index
                95c107
                < public SolrClientImpl(URL baseURL) throws Exception

                > public SolrClientImpl(URL baseURL, boolean allowDups) throws Exception
                103c115
                <

                > this.allowDups = allowDups;
                243c255,260
                < writer.write("<add>");

                > StringBuffer strAdd = new StringBuffer("<add ");
                > if (allowDups == true)

                { > strAdd.append("allowDups=\"true\""); > }

                > strAdd.append(">");
                > writer.write(strAdd.toString());

              • SolrClientImpl.java.patch ***
          Show
          Ben Incani added a comment - The SolrClientImpl does not implement the following optional attributes for "add" as documented in http://wiki.apache.org/solr/UpdateXmlMessages allowDups = "true" | "false" — default is "false" overwritePending = "true" | "false" — default is negation of allowDups overwriteCommitted = "true"|"false" — default is negation of allowDups Attached is patch for SolrClientImpl.java which implements allowDups. SolrClientImpl.java.patch *** 48a49,55 > > /** > * Optional attributes for "add" > */ > protected boolean allowDups; > protected boolean overwritePending; // TODO: not implemented > protected boolean overwriteCommitted; // TODO: not implemented 86a94,97 > public SolrClientImpl(URL baseURL) throws Exception { > this(baseURL, false); > } > 91a103 > * @param allowDups allow duplicates to be added to the index 95c107 < public SolrClientImpl(URL baseURL) throws Exception — > public SolrClientImpl(URL baseURL, boolean allowDups) throws Exception 103c115 < — > this.allowDups = allowDups; 243c255,260 < writer.write("<add>"); — > StringBuffer strAdd = new StringBuffer("<add "); > if (allowDups == true) { > strAdd.append("allowDups=\"true\""); > } > strAdd.append(">"); > writer.write(strAdd.toString()); SolrClientImpl.java.patch ***
          Hide
          Ryan McKinley added a comment -

          Hi Ben-

          Thanks for the patch. Will Johnson added the options to the java client on:
          http://solrstuff.org/svn/solrj/

          This implementation is now quite stable (thanks to lots of help from Will!) and I hope will be integrated into solr trunk shortly after 1.2 (this week?!)

          About the allowDups/overwritePending/overwriteCommited options... what part of them do you use? In SOLR-60, there is talk of replacing the three options with a (simpler) overwrite=true/false – maybe we should change the client api to:

          UpdateResponse add( SolrDocument doc, boolean overwrite ) throws SolrServerException;

          In anticipation of this change?

          Show
          Ryan McKinley added a comment - Hi Ben- Thanks for the patch. Will Johnson added the options to the java client on: http://solrstuff.org/svn/solrj/ This implementation is now quite stable (thanks to lots of help from Will!) and I hope will be integrated into solr trunk shortly after 1.2 (this week?!) About the allowDups/overwritePending/overwriteCommited options... what part of them do you use? In SOLR-60 , there is talk of replacing the three options with a (simpler) overwrite=true/false – maybe we should change the client api to: UpdateResponse add( SolrDocument doc, boolean overwrite ) throws SolrServerException; In anticipation of this change?
          Hide
          Walter Ferrara added a comment - - edited

          [I'm new to solrj, so everything I'm writing can be useless]

          While trying to execute range query, using this query:
          Text:Hello +Date:[1895011 TO 18971128]
          [jdk 1.6/netbeans 5.5/solr1.2/solrj revision 125]

          I kept getting IllegalArgumentException:
          [...]
          Caused by: java.lang.IllegalArgumentException: Invalid uri 'http://localhost:8983/solr/select?q=Text:Hello+%2BDate:[1895011+TO+18971128]&fl=ID,Date,score&rows=800&wt=xml&version=2.2': Invalid query
          at org.apache.commons.httpclient.HttpMethodBase.<init>(HttpMethodBase.java:222)
          at org.apache.commons.httpclient.methods.GetMethod.<init>(GetMethod.java:89)
          at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:108)
          [...]
          It seems related to un-escaped '[' and ']'.

          the solution was to patch StrUtils.java, in order to encode square brackets.
          If it is of any use, I'm attaching the patch
          – Walter

          ====
          StrUtils.java.patch

          204,207d203
          < case '[': dest.append("%5B"); break; // to allow range query, 20070706 (w)
          < case ']': dest.append("%5D"); break;
          < case '

          {': dest.append("%7B"); break; < case '}

          ': dest.append("%7D"); break;

          Show
          Walter Ferrara added a comment - - edited [I'm new to solrj, so everything I'm writing can be useless] While trying to execute range query, using this query: Text:Hello +Date: [1895011 TO 18971128] [jdk 1.6/netbeans 5.5/solr1.2/solrj revision 125] I kept getting IllegalArgumentException: [...] Caused by: java.lang.IllegalArgumentException: Invalid uri 'http://localhost:8983/solr/select?q=Text:Hello+%2BDate: [1895011+TO+18971128] &fl=ID,Date,score&rows=800&wt=xml&version=2.2': Invalid query at org.apache.commons.httpclient.HttpMethodBase.<init>(HttpMethodBase.java:222) at org.apache.commons.httpclient.methods.GetMethod.<init>(GetMethod.java:89) at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:108) [...] It seems related to un-escaped ' [' and '] '. the solution was to patch StrUtils.java, in order to encode square brackets. If it is of any use, I'm attaching the patch – Walter ==== StrUtils.java.patch 204,207d203 < case '[': dest.append("%5B"); break; // to allow range query, 20070706 (w) < case ']': dest.append("%5D"); break; < case ' {': dest.append("%7B"); break; < case '} ': dest.append("%7D"); break;
          Hide
          Ryan McKinley added a comment -

          Hi Walter-

          I just updated http://solrstuff.org/svn/solrj/ to use:

          URLEncoder.encode( val, "UTF-8" )

          rather then:

          StrUtils.partialURLEncodeVal( val )

          Give it a try and let me know if you have problems... (i have done date range queries successfully with resin/jetty, but netbeans must be different!)

          Show
          Ryan McKinley added a comment - Hi Walter- I just updated http://solrstuff.org/svn/solrj/ to use: URLEncoder.encode( val, "UTF-8" ) rather then: StrUtils.partialURLEncodeVal( val ) Give it a try and let me know if you have problems... (i have done date range queries successfully with resin/jetty, but netbeans must be different!)
          Hide
          Yonik Seeley added a comment -

          FYI partialURLEncodeVal was meant for readable, yet unambiguous logging... hence a minimum of escaping is done (but enough to easily paste into a browser and let it do the rest of the escaping when you sumbit).

          Show
          Yonik Seeley added a comment - FYI partialURLEncodeVal was meant for readable, yet unambiguous logging... hence a minimum of escaping is done (but enough to easily paste into a browser and let it do the rest of the escaping when you sumbit).
          Hide
          Walter Ferrara added a comment -

          Latest rev works perfectly thanks.
          I've been making some time test with this client (only searching), and overall results show high times: this maybe due to my minimal knowledge on solr, but solr seems fast, is data-receiving/parsing on client that seems slow. Even when solr report 0ms (due to cache it I presume) it still take 200ms to get results (QueryResponse .getElapsedTime()). I'm using this code:

          SolrQuery query = new SolrQuery(queryString);
          CommonsHttpSolrServer server = new CommonsHttpSolrServer("http://localhost:8983/solr/");
          QueryResponse response = server.query(query);
          SolrDocumentList list = response.getResults();

          The slowdown seems to be in client.executeMethod(method) (CommonsHttpSolrServer) Any way to speed up (assuming I'm not totally wrong on how to use this client...)? Reusing same http connection for multiple queries? Playing with MultiThreadedHttpConnectionManager helped a bit, but doesn't seems the solution

          Show
          Walter Ferrara added a comment - Latest rev works perfectly thanks. I've been making some time test with this client (only searching), and overall results show high times: this maybe due to my minimal knowledge on solr, but solr seems fast, is data-receiving/parsing on client that seems slow. Even when solr report 0ms (due to cache it I presume) it still take 200ms to get results (QueryResponse .getElapsedTime()). I'm using this code: SolrQuery query = new SolrQuery(queryString); CommonsHttpSolrServer server = new CommonsHttpSolrServer("http://localhost:8983/solr/"); QueryResponse response = server.query(query); SolrDocumentList list = response.getResults(); The slowdown seems to be in client.executeMethod(method) (CommonsHttpSolrServer) Any way to speed up (assuming I'm not totally wrong on how to use this client...)? Reusing same http connection for multiple queries? Playing with MultiThreadedHttpConnectionManager helped a bit, but doesn't seems the solution
          Hide
          Ryan McKinley added a comment -

          I don't know if you are on solr-dev, Yonik noted that the QTime does not include the time to write the response, only the query time. To get an accurate number for how long the whole query takes, check your app server logs
          http://www.nabble.com/Re%3A-A-simple-Java-client-for-updating-and-searching-tf3890950.html

          To get a quick response from solr, try rows=0 or a 404 path. (Of course, the speed will depend on you network connection speed between client-server)

          Show
          Ryan McKinley added a comment - I don't know if you are on solr-dev, Yonik noted that the QTime does not include the time to write the response, only the query time. To get an accurate number for how long the whole query takes, check your app server logs http://www.nabble.com/Re%3A-A-simple-Java-client-for-updating-and-searching-tf3890950.html To get a quick response from solr, try rows=0 or a 404 path. (Of course, the speed will depend on you network connection speed between client-server)
          Hide
          Ryan McKinley added a comment -

          I'm integrating SOLR-20 with trunk now...

          The basic stuff is no problem. I'm struggling with the best way to:

          • included shared libraries (commons-io, etc)
          • structure build.xml
          • add tests that launch jetty
          • package solrj (and dependencies)

          I think the best approach is to commit my best effort and then have you
          all review/fix/modify. Building a useful patch is unrealistic.

          ryan

          Show
          Ryan McKinley added a comment - I'm integrating SOLR-20 with trunk now... The basic stuff is no problem. I'm struggling with the best way to: included shared libraries (commons-io, etc) structure build.xml add tests that launch jetty package solrj (and dependencies) I think the best approach is to commit my best effort and then have you all review/fix/modify. Building a useful patch is unrealistic. ryan
          Hide
          Michael Young added a comment -

          We are planning to replace our custom Lucene implementation with Solr in the next release of Liferay. This Java client would be extremely useful to us and we would like to see it in the next stable release. When do you anticipate this, or at least an alpha version?

          Show
          Michael Young added a comment - We are planning to replace our custom Lucene implementation with Solr in the next release of Liferay. This Java client would be extremely useful to us and we would like to see it in the next stable release. When do you anticipate this, or at least an alpha version?
          Hide
          Ryan McKinley added a comment -

          solr 1.2 was released ~1 week ago so the next official stable release is at least a few months out.

          The solrj client is quite stable (I won't say that too strongly until more people are using it) and will be included in solr nightly builds. While I don't recommend using the solr server nightly builds, the client should be ok.

          Show
          Ryan McKinley added a comment - solr 1.2 was released ~1 week ago so the next official stable release is at least a few months out. The solrj client is quite stable (I won't say that too strongly until more people are using it) and will be included in solr nightly builds. While I don't recommend using the solr server nightly builds, the client should be ok.
          Hide
          Ryan McKinley added a comment -

          Added to trunk... any new problems should get their own issue.

          Show
          Ryan McKinley added a comment - Added to trunk... any new problems should get their own issue.
          Hide
          Hoss Man added a comment -

          This bug was modified as part of a bulk update using the criteria...

          • Marked "Resolved" and "Fixed"
          • Had no "Fix Version" versions
          • Was listed in the CHANGES.txt for 1.3 as of today 2008-03-15

          The Fix Version for all 29 issues found was set to 1.3, email notification was suppressed to prevent excessive email.

          For a list of all the issues modified, search jira comments for this (hopefully) unique string: batch20070315hossman1

          Show
          Hoss Man added a comment - This bug was modified as part of a bulk update using the criteria... Marked "Resolved" and "Fixed" Had no "Fix Version" versions Was listed in the CHANGES.txt for 1.3 as of today 2008-03-15 The Fix Version for all 29 issues found was set to 1.3, email notification was suppressed to prevent excessive email. For a list of all the issues modified, search jira comments for this (hopefully) unique string: batch20070315hossman1

            People

            • Assignee:
              Ryan McKinley
              Reporter:
              Darren Erik Vengroff
            • Votes:
              8 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development