Solr
  1. Solr
  2. SOLR-1038

Enhance CommonsHttpSolrServer to add docs in batch

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.4
    • Component/s: clients - java
    • Labels:
      None

      Description

      add two methods to ComonsHttpSolrServer to facilitate batch upload of docs

      public void addAndCommit(Iterator<SolrInputDocument> docIter , SolrParams commitParams)  
      throws SolrServerException, IOException 
      

      This helps users to upload all docs in a single http request

      1. SOLR-1038.patch
        7 kB
        Noble Paul
      2. SOLR-1038.patch
        13 kB
        Noble Paul
      3. SOLR-1038.patch
        2 kB
        Noble Paul
      4. SOLR-1038.patch
        3 kB
        Noble Paul

        Activity

        Hide
        Noble Paul added a comment -

        the fix .
        But no testcase yet. So not yet ripe to be committed

        Show
        Noble Paul added a comment - the fix . But no testcase yet. So not yet ripe to be committed
        Hide
        Noble Paul added a comment -

        with a testcase and a bug fix.

        Show
        Noble Paul added a comment - with a testcase and a bug fix.
        Hide
        Shalin Shekhar Mangar added a comment -

        I see some changes related to writing Map in JavaBinUpdateRequestCodec. What is that used for?

        Show
        Shalin Shekhar Mangar added a comment - I see some changes related to writing Map in JavaBinUpdateRequestCodec. What is that used for?
        Hide
        Noble Paul added a comment -

        earlier, all the SolrInputDocument we converted in advance to the collection objects . It is not possible now because we support an iterator. So the conversion is done just in time

        Show
        Noble Paul added a comment - earlier, all the SolrInputDocument we converted in advance to the collection objects . It is not possible now because we support an iterator. So the conversion is done just in time
        Hide
        Shalin Shekhar Mangar added a comment -

        Ok, understood.

        How about we also add another method:

        public void addAndCommit(Iterator<SolrInputDocument> docIter , SolrParams commitParams, boolean rollbackOnError) 
        throws SolrServerException, IOException
        
        Show
        Shalin Shekhar Mangar added a comment - Ok, understood. How about we also add another method: public void addAndCommit(Iterator<SolrInputDocument> docIter , SolrParams commitParams, boolean rollbackOnError) throws SolrServerException, IOException
        Hide
        Noble Paul added a comment -

        makes sense

        Show
        Noble Paul added a comment - makes sense
        Hide
        Shalin Shekhar Mangar added a comment -

        OK, I had misunderstood the feature. I thought it was async. In this case, the new method is not necessary. A client can catch an exception and call rollback themselves.

        I'll commit this shortly.

        Show
        Shalin Shekhar Mangar added a comment - OK, I had misunderstood the feature. I thought it was async. In this case, the new method is not necessary. A client can catch an exception and call rollback themselves. I'll commit this shortly.
        Hide
        Shalin Shekhar Mangar added a comment -

        Committed revision 755882.

        Thanks Noble!

        Show
        Shalin Shekhar Mangar added a comment - Committed revision 755882. Thanks Noble!
        Hide
        Noble Paul added a comment - - edited

        The add and commit is slightly inconvenient if I wish to do update in multiple threads and comit all at once.
        So a simple add(Iterator<SolrInputDocument> ) method should be enough. commmit/rollback can be done by the user after the add is done

        moreover if I wish to do a rollback if something goes wrong in between, I do not have the liberty to do so.

        Show
        Noble Paul added a comment - - edited The add and commit is slightly inconvenient if I wish to do update in multiple threads and comit all at once. So a simple add(Iterator<SolrInputDocument> ) method should be enough. commmit/rollback can be done by the user after the add is done moreover if I wish to do a rollback if something goes wrong in between, I do not have the liberty to do so.
        Hide
        Noble Paul added a comment -

        the changes

        Show
        Noble Paul added a comment - the changes
        Hide
        Shalin Shekhar Mangar added a comment -

        The add and commit is slightly inconvenient if I wish to do update in multiple threads and comit all at once.

        Makes sense.

        Committed revision 760113.

        Show
        Shalin Shekhar Mangar added a comment - The add and commit is slightly inconvenient if I wish to do update in multiple threads and comit all at once. Makes sense. Committed revision 760113.
        Hide
        Noble Paul added a comment -

        added a new method

         /**
           * Adds the beans supplied by the given iterator.
           *
           * @param beanIterator  the iterator which returns Beans
           *
           * @return the response from the SolrServer
           */
          public UpdateResponse addBeans(final Iterator<?> beanIterator)
                  throws SolrServerException, IOException 
        
        Show
        Noble Paul added a comment - added a new method /** * Adds the beans supplied by the given iterator. * * @param beanIterator the iterator which returns Beans * * @ return the response from the SolrServer */ public UpdateResponse addBeans( final Iterator<?> beanIterator) throws SolrServerException, IOException
        Hide
        Shalin Shekhar Mangar added a comment -

        Committed revision 762176.

        Thanks Noble!

        Show
        Shalin Shekhar Mangar added a comment - Committed revision 762176. Thanks Noble!
        Hide
        Grant Ingersoll added a comment -

        Bulk close for Solr 1.4

        Show
        Grant Ingersoll added a comment - Bulk close for Solr 1.4

          People

          • Assignee:
            Shalin Shekhar Mangar
            Reporter:
            Noble Paul
          • Votes:
            1 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development