Solr
  1. Solr
  2. SOLR-212

Embeddable class to call solr directly

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.2
    • Component/s: None
    • Labels:
      None

      Description

      For some embedded applications, it is useful to call solr without running an HTTP server. This class mimics the behavior you would get if you sent the request through an HTTP connection. It is designed to work nicely (ie simple) with JNI

      the main function is:

      public class DirectSolrConnection
      {
      String request( String pathAndParams, String body ) throws Exception

      { ... }

      }

      1. DirectSolrConnection.java
        7 kB
        damu liu
      2. embeddedSolr.zip
        1.28 MB
        Brian Whitman
      3. SOLR-212-DirectSolrConnection.patch
        11 kB
        Ryan McKinley
      4. SOLR-212-DirectSolrConnection.patch
        9 kB
        Ryan McKinley
      5. SOLR-212-DirectSolrConnection.patch
        8 kB
        Ryan McKinley
      6. SOLR-212-DirectSolrConnection.patch
        7 kB
        Ryan McKinley
      7. SOLR-212-DirectSolrConnection.patch
        6 kB
        Ryan McKinley

        Activity

        Hide
        Ryan McKinley added a comment -

        This class now sits in: o.a.s.servlet because it uses package protected request parsing functions.

        It has a really simple test that should be extended.

        Example usage:

        DirectSolrConnection solr = new DirectSolrConnection();
        String json = solr.request( "/select?qt=dismax&wt=json&q=...", null );
        String xml = solr.request( "/update", "<add><doc><field ..." );

        Show
        Ryan McKinley added a comment - This class now sits in: o.a.s.servlet because it uses package protected request parsing functions. It has a really simple test that should be extended. Example usage: DirectSolrConnection solr = new DirectSolrConnection(); String json = solr.request( "/select?qt=dismax&wt=json&q=...", null ); String xml = solr.request( "/update", "<add><doc><field ..." );
        Hide
        Otis Gospodnetic added a comment -

        Mucho cool. I see the DirectSolrConnection.coffee doesn't have ASL v2 in it.
        How does this relate to the code I see on http://wiki.apache.org/solr/EmbeddedSolr ?
        Typo: initalized -> initialized (in javadoc + code)

        Show
        Otis Gospodnetic added a comment - Mucho cool. I see the DirectSolrConnection.coffee doesn't have ASL v2 in it. How does this relate to the code I see on http://wiki.apache.org/solr/EmbeddedSolr ? Typo: initalized -> initialized (in javadoc + code)
        Hide
        Ryan McKinley added a comment -

        Added header and fixed "Initialized" typo
        (fixed the typo in Config r533207)

        This approach differs from the one outlined in http://wiki.apache.org/solr/EmbeddedSolr in that it is strictly a text interface that mimics the request and response stream. The other approach gives you direct access to the lucene Document class. The text approach is nice because it means the same code would work if it needed to run across a network.

        Show
        Ryan McKinley added a comment - Added header and fixed "Initialized" typo (fixed the typo in Config r533207) This approach differs from the one outlined in http://wiki.apache.org/solr/EmbeddedSolr in that it is strictly a text interface that mimics the request and response stream. The other approach gives you direct access to the lucene Document class. The text approach is nice because it means the same code would work if it needed to run across a network.
        Hide
        Ryan McKinley added a comment -

        Adding dataDir to an optional constructor.

        Show
        Ryan McKinley added a comment - Adding dataDir to an optional constructor.
        Hide
        Brian Whitman added a comment -

        Much love from user land on this one. I just successfully put solr in a C app without any webserver running using JNI.

        After I clean up my JNI calling code I can post an example app here to show how it's done on the client side if anyone is interested?

        Show
        Brian Whitman added a comment - Much love from user land on this one. I just successfully put solr in a C app without any webserver running using JNI. After I clean up my JNI calling code I can post an example app here to show how it's done on the client side if anyone is interested?
        Hide
        Brian Whitman added a comment -

        Since the main use case of SOLR-212 is to embed it in client applications, we should be careful about logging. As of now SOLR-212 will spit stuff all over stderr.

        I suggest putting this

        System.setProperty("java.util.logging.config.file", instanceDir+"/conf/logging.properties");

        near line 79 of DirectSolrConnection.java. That way, if a developer/user chooses, they can put a logging.prop file in conf and set direct logging of Solr requests either to their own application logs or a file. If the conf/logging.properties file does not exist, I believe the default logging.properties will be used (which is what happens now.)

        Show
        Brian Whitman added a comment - Since the main use case of SOLR-212 is to embed it in client applications, we should be careful about logging. As of now SOLR-212 will spit stuff all over stderr. I suggest putting this System.setProperty("java.util.logging.config.file", instanceDir+"/conf/logging.properties"); near line 79 of DirectSolrConnection.java. That way, if a developer/user chooses, they can put a logging.prop file in conf and set direct logging of Solr requests either to their own application logs or a file. If the conf/logging.properties file does not exist, I believe the default logging.properties will be used (which is what happens now.)
        Hide
        Ryan McKinley added a comment -

        Updated to take an (optional) logging path

        Show
        Ryan McKinley added a comment - Updated to take an (optional) logging path
        Hide
        Otis Gospodnetic added a comment -

        Brian: interested!

        Show
        Otis Gospodnetic added a comment - Brian: interested!
        Hide
        Ryan McKinley added a comment -

        Added some real tests (it adds/delete/searches for documents) and fixed the things that broke

        Show
        Ryan McKinley added a comment - Added some real tests (it adds/delete/searches for documents) and fixed the things that broke
        Hide
        Ryan McKinley added a comment -

        added in rev 533538

        Yes Brian, we'd love to see the JNI code!

        Show
        Ryan McKinley added a comment - added in rev 533538 Yes Brian, we'd love to see the JNI code!
        Hide
        Brian Whitman added a comment -

        Am working on extracting it from Cocoa as we speak... watch this space!

        Show
        Brian Whitman added a comment - Am working on extracting it from Cocoa as we speak... watch this space!
        Hide
        Brian Whitman added a comment -

        I didn't have time to extract it from Cocoa/ObjC. Here is the xcode project with everything anyway – so right now you'll need OSX to try this out. This is a very simple test of Ryan's SOLR-212 patch, it queries, adds a document and commits, all without a web server! Let me know if you have any questions. --brian

        Show
        Brian Whitman added a comment - I didn't have time to extract it from Cocoa/ObjC. Here is the xcode project with everything anyway – so right now you'll need OSX to try this out. This is a very simple test of Ryan's SOLR-212 patch, it queries, adds a document and commits, all without a web server! Let me know if you have any questions. --brian
        Hide
        J.J. Larrea added a comment -

        One issue which comes up both DirectSolrConnection.java and the http://wiki.apache.org/solr/EmbeddedSolr technique (which I have started using for a custom indexer) is that the SolrCore constructor will always create an initial searcher, and doing so will execute any warming SolrEventListeners defined in the config, whether or not the application needs it. Besides delaying startup and wasting cycles, it can cause the allocation of significant amounts of memory for sort FieldCaches, Solr caches, and the like, none of which will be used for a non-search application. And

        Other than maintaining 2 sets of config files differening only in whether warmup queries are defined, does anyone have any ideas for how warmup can be selectively enabled based on the calling context?

        Show
        J.J. Larrea added a comment - One issue which comes up both DirectSolrConnection.java and the http://wiki.apache.org/solr/EmbeddedSolr technique (which I have started using for a custom indexer) is that the SolrCore constructor will always create an initial searcher, and doing so will execute any warming SolrEventListeners defined in the config, whether or not the application needs it. Besides delaying startup and wasting cycles, it can cause the allocation of significant amounts of memory for sort FieldCaches, Solr caches, and the like, none of which will be used for a non-search application. And Other than maintaining 2 sets of config files differening only in whether warmup queries are defined, does anyone have any ideas for how warmup can be selectively enabled based on the calling context?
        Hide
        damu liu added a comment -

        The DirectSolrConnection class should have an additional method added:

        public void clearup()

        { core.close(); }

        otherwise, the program cannot stop, there are searcher and listeners running in other thread. Use this method to clear the resources.

        Show
        damu liu added a comment - The DirectSolrConnection class should have an additional method added: public void clearup() { core.close(); } otherwise, the program cannot stop, there are searcher and listeners running in other thread. Use this method to clear the resources.
        Hide
        damu liu added a comment -

        My version of DirectSolrConnection with clearup method added.

        Show
        damu liu added a comment - My version of DirectSolrConnection with clearup method added.
        Hide
        damu liu added a comment -

        With clear up method added to clear up used resources

        Show
        damu liu added a comment - With clear up method added to clear up used resources
        Hide
        Ryan McKinley added a comment -

        I added a "close()" method in rev 561922

        Show
        Ryan McKinley added a comment - I added a "close()" method in rev 561922
        Hide
        damu liu added a comment -

        That is great!

        Show
        damu liu added a comment - That is great!
        Hide
        zhang.zuxin added a comment - - edited

        Expect standard C/C++ versions with a solr plgu-in!

        Show
        zhang.zuxin added a comment - - edited Expect standard C/C++ versions with a solr plgu-in!

          People

          • Assignee:
            Ryan McKinley
            Reporter:
            Ryan McKinley
          • Votes:
            2 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development