Solr
  1. Solr
  2. SOLR-865

Support document updates in binary format

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.4
    • Fix Version/s: 1.4
    • Component/s: clients - java, update
    • Labels:
      None

      Description

      SolrJ uses the fast and compact 'javabin' format for querying Solr which is a huge performance improvement over the xml format. We should make it possible to use the 'javabin' format for document updates as well.

      1. SOLR-865.patch
        29 kB
        Shalin Shekhar Mangar
      2. SOLR-865.patch
        25 kB
        Shalin Shekhar Mangar
      3. SOLR-865.patch
        25 kB
        Noble Paul
      4. SOLR-865.patch
        26 kB
        Noble Paul
      5. SOLR-865.patch
        26 kB
        Noble Paul
      6. SOLR-865.patch
        13 kB
        Noble Paul
      7. SOLR-865.patch
        16 kB
        Noble Paul

        Activity

        Hide
        Noble Paul added a comment - - edited
        • A new UpdateHandler called BinaryUpdateRequestHandler
        • A codec for UpdateRequest. The codec does streaming for unmarshalling
        • RequestWriter is pluggable for CommonsHttpSolrServer. default is xml and binary format can be used by server.setRequestWriter(new BinaryRequestWriter());
        • A testcase added LargeVolumeBinaryJettyTest It fails saying http://localhost:4020/example/update/javabin canot be found . has to be fixed.
        Show
        Noble Paul added a comment - - edited A new UpdateHandler called BinaryUpdateRequestHandler A codec for UpdateRequest. The codec does streaming for unmarshalling RequestWriter is pluggable for CommonsHttpSolrServer. default is xml and binary format can be used by server.setRequestWriter(new BinaryRequestWriter()); A testcase added LargeVolumeBinaryJettyTest It fails saying http://localhost:4020/example/update/javabin canot be found . has to be fixed.
        Hide
        Noble Paul added a comment -

        Last patch was incomplete

        Thie testcase fails with the following mesage

        exceeded limit of maxWarmingSearchers=2, try again later.
        Jan 5, 2009 5:44:40 PM org.apache.solr.update.processor.LogUpdateProcessor finish
        INFO: {} 0 15
        Jan 5, 2009 5:44:40 PM org.apache.solr.common.SolrException log
        SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try again later.
        	at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1050)
        	at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:350)
        	at org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:78)
        	at org.apache.solr.handler.BinaryUpdateRequestHandler.parseAndLoadDocs(BinaryUpdateRequestHandler.java:95)
        	at 
        

        any idea

        Show
        Noble Paul added a comment - Last patch was incomplete Thie testcase fails with the following mesage exceeded limit of maxWarmingSearchers=2, try again later. Jan 5, 2009 5:44:40 PM org.apache.solr.update.processor.LogUpdateProcessor finish INFO: {} 0 15 Jan 5, 2009 5:44:40 PM org.apache.solr.common.SolrException log SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try again later. at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1050) at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:350) at org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:78) at org.apache.solr.handler.BinaryUpdateRequestHandler.parseAndLoadDocs(BinaryUpdateRequestHandler.java:95) at any idea
        Hide
        Noble Paul added a comment -

        Last patch missed a few files

        Show
        Noble Paul added a comment - Last patch missed a few files
        Hide
        Noble Paul added a comment -

        new patch. but still same error

        Show
        Noble Paul added a comment - new patch. but still same error
        Hide
        Noble Paul added a comment -

        Tests pass

        Show
        Noble Paul added a comment - Tests pass
        Hide
        Shalin Shekhar Mangar added a comment -
        1. Fixed failure in SolrInfoMBean test by adding the version, source to BinaryUpdateRequestHandler
        2. Added Javadocs

        It would be nice to get some feedback on the RequestWriter API. We should also think about removing the XML writing part from inside UpdateRequest to an XmlUpdateRequestWriter. StreamingUpdateSolrServer should also move to depend on the request writer API.

        Show
        Shalin Shekhar Mangar added a comment - Fixed failure in SolrInfoMBean test by adding the version, source to BinaryUpdateRequestHandler Added Javadocs It would be nice to get some feedback on the RequestWriter API. We should also think about removing the XML writing part from inside UpdateRequest to an XmlUpdateRequestWriter. StreamingUpdateSolrServer should also move to depend on the request writer API.
        Hide
        Shalin Shekhar Mangar added a comment -

        Syncing with trunk and adding javadocs in a few places.

        I'll commit this shortly. We can open separate issues to migrate other SolrServer implementations to the RequestWriter API.

        Show
        Shalin Shekhar Mangar added a comment - Syncing with trunk and adding javadocs in a few places. I'll commit this shortly. We can open separate issues to migrate other SolrServer implementations to the RequestWriter API.
        Hide
        Shalin Shekhar Mangar added a comment -

        Committed revision 734796.

        Thanks Noble!

        Show
        Shalin Shekhar Mangar added a comment - Committed revision 734796. Thanks Noble!
        Hide
        Grant Ingersoll added a comment -

        Bulk close Solr 1.4 issues

        Show
        Grant Ingersoll added a comment - Bulk close Solr 1.4 issues

          People

          • Assignee:
            Shalin Shekhar Mangar
            Reporter:
            Shalin Shekhar Mangar
          • Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development