Solr
  1. Solr
  2. SOLR-1038

Enhance CommonsHttpSolrServer to add docs in batch

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.4
    • Component/s: clients - java
    • Labels:
      None

      Description

      add two methods to ComonsHttpSolrServer to facilitate batch upload of docs

      public void addAndCommit(Iterator<SolrInputDocument> docIter , SolrParams commitParams)  
      throws SolrServerException, IOException 
      

      This helps users to upload all docs in a single http request

      1. SOLR-1038.patch
        3 kB
        Noble Paul
      2. SOLR-1038.patch
        2 kB
        Noble Paul
      3. SOLR-1038.patch
        13 kB
        Noble Paul
      4. SOLR-1038.patch
        7 kB
        Noble Paul

        Activity

        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Resolved Resolved
        21d 16h 36m 1 Shalin Shekhar Mangar 19/Mar/09 09:15
        Resolved Resolved Reopened Reopened
        11d 1h 13m 1 Noble Paul 30/Mar/09 11:29
        Reopened Reopened Resolved Resolved
        8h 51m 1 Shalin Shekhar Mangar 30/Mar/09 20:20
        Resolved Resolved Closed Closed
        224d 20h 31m 1 Grant Ingersoll 10/Nov/09 15:51
        Grant Ingersoll made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Hide
        Grant Ingersoll added a comment -

        Bulk close for Solr 1.4

        Show
        Grant Ingersoll added a comment - Bulk close for Solr 1.4
        Hide
        Shalin Shekhar Mangar added a comment -

        Committed revision 762176.

        Thanks Noble!

        Show
        Shalin Shekhar Mangar added a comment - Committed revision 762176. Thanks Noble!
        Noble Paul made changes -
        Attachment SOLR-1038.patch [ 12404629 ]
        Hide
        Noble Paul added a comment -

        added a new method

         /**
           * Adds the beans supplied by the given iterator.
           *
           * @param beanIterator  the iterator which returns Beans
           *
           * @return the response from the SolrServer
           */
          public UpdateResponse addBeans(final Iterator<?> beanIterator)
                  throws SolrServerException, IOException 
        
        Show
        Noble Paul added a comment - added a new method /** * Adds the beans supplied by the given iterator. * * @param beanIterator the iterator which returns Beans * * @ return the response from the SolrServer */ public UpdateResponse addBeans( final Iterator<?> beanIterator) throws SolrServerException, IOException
        Shalin Shekhar Mangar made changes -
        Status Reopened [ 4 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        Shalin Shekhar Mangar added a comment -

        The add and commit is slightly inconvenient if I wish to do update in multiple threads and comit all at once.

        Makes sense.

        Committed revision 760113.

        Show
        Shalin Shekhar Mangar added a comment - The add and commit is slightly inconvenient if I wish to do update in multiple threads and comit all at once. Makes sense. Committed revision 760113.
        Noble Paul made changes -
        Attachment SOLR-1038.patch [ 12404119 ]
        Hide
        Noble Paul added a comment -

        the changes

        Show
        Noble Paul added a comment - the changes
        Noble Paul made changes -
        Resolution Fixed [ 1 ]
        Status Resolved [ 5 ] Reopened [ 4 ]
        Hide
        Noble Paul added a comment - - edited

        The add and commit is slightly inconvenient if I wish to do update in multiple threads and comit all at once.
        So a simple add(Iterator<SolrInputDocument> ) method should be enough. commmit/rollback can be done by the user after the add is done

        moreover if I wish to do a rollback if something goes wrong in between, I do not have the liberty to do so.

        Show
        Noble Paul added a comment - - edited The add and commit is slightly inconvenient if I wish to do update in multiple threads and comit all at once. So a simple add(Iterator<SolrInputDocument> ) method should be enough. commmit/rollback can be done by the user after the add is done moreover if I wish to do a rollback if something goes wrong in between, I do not have the liberty to do so.
        Shalin Shekhar Mangar made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        Shalin Shekhar Mangar added a comment -

        Committed revision 755882.

        Thanks Noble!

        Show
        Shalin Shekhar Mangar added a comment - Committed revision 755882. Thanks Noble!
        Hide
        Shalin Shekhar Mangar added a comment -

        OK, I had misunderstood the feature. I thought it was async. In this case, the new method is not necessary. A client can catch an exception and call rollback themselves.

        I'll commit this shortly.

        Show
        Shalin Shekhar Mangar added a comment - OK, I had misunderstood the feature. I thought it was async. In this case, the new method is not necessary. A client can catch an exception and call rollback themselves. I'll commit this shortly.
        Hide
        Noble Paul added a comment -

        makes sense

        Show
        Noble Paul added a comment - makes sense
        Hide
        Shalin Shekhar Mangar added a comment -

        Ok, understood.

        How about we also add another method:

        public void addAndCommit(Iterator<SolrInputDocument> docIter , SolrParams commitParams, boolean rollbackOnError) 
        throws SolrServerException, IOException
        
        Show
        Shalin Shekhar Mangar added a comment - Ok, understood. How about we also add another method: public void addAndCommit(Iterator<SolrInputDocument> docIter , SolrParams commitParams, boolean rollbackOnError) throws SolrServerException, IOException
        Hide
        Noble Paul added a comment -

        earlier, all the SolrInputDocument we converted in advance to the collection objects . It is not possible now because we support an iterator. So the conversion is done just in time

        Show
        Noble Paul added a comment - earlier, all the SolrInputDocument we converted in advance to the collection objects . It is not possible now because we support an iterator. So the conversion is done just in time
        Hide
        Shalin Shekhar Mangar added a comment -

        I see some changes related to writing Map in JavaBinUpdateRequestCodec. What is that used for?

        Show
        Shalin Shekhar Mangar added a comment - I see some changes related to writing Map in JavaBinUpdateRequestCodec. What is that used for?
        Noble Paul made changes -
        Description add two methods to ComonsHttpSolrServer to facilitate batch upload of docs
        {code:java}
        public void addAndCommit(Iterator<SolrInputDocument> docIter , boolean async, Properties commitParams)
        throws SolrServerException, IOException

        public void addAndCommit(Iterator<Object> beanIter , boolean async , Properties commitParams)
        throws SolrServerException, IOException
        {code}

        This helps users to upload all docs in a single http request
        add two methods to ComonsHttpSolrServer to facilitate batch upload of docs
        {code:java}
        public void addAndCommit(Iterator<SolrInputDocument> docIter , SolrParams commitParams)
        throws SolrServerException, IOException
        {code}

        This helps users to upload all docs in a single http request
        Shalin Shekhar Mangar made changes -
        Assignee Shalin Shekhar Mangar [ shalinmangar ]
        Noble Paul made changes -
        Attachment SOLR-1038.patch [ 12402456 ]
        Hide
        Noble Paul added a comment -

        with a testcase and a bug fix.

        Show
        Noble Paul added a comment - with a testcase and a bug fix.
        Noble Paul made changes -
        Fix Version/s 1.4 [ 12313351 ]
        Fix Version/s 1.5 [ 12313566 ]
        Noble Paul made changes -
        Attachment SOLR-1038.patch [ 12401805 ]
        Hide
        Noble Paul added a comment -

        the fix .
        But no testcase yet. So not yet ripe to be committed

        Show
        Noble Paul added a comment - the fix . But no testcase yet. So not yet ripe to be committed
        Noble Paul made changes -
        Field Original Value New Value
        Component/s clients - java [ 12311580 ]
        Component/s clients - C# [ 12311816 ]
        Fix Version/s 1.5 [ 12313566 ]
        Noble Paul created issue -

          People

          • Assignee:
            Shalin Shekhar Mangar
            Reporter:
            Noble Paul
          • Votes:
            1 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development