Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-10676

Optimize the reindexing of sunspot solr

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Invalid
    • Affects Version/s: 5.0
    • Fix Version/s: None
    • Component/s: clients - ruby - flare
    • Labels:
      None

      Description

      We are using solr 5.0. <luceneMatchVersion>5.0.0</luceneMatchVersion>
      We have more than 5 million products. It is taking around 3.30 hours to reindex all the products.

      For optimizing the reindexing speed, we have used the following configurations

      <indexConfig>
      <ramBufferSizeMB>960</ramBufferSizeMB>
      <mergePolicyFactory>100</mergePolicyFactory>
      <mergeScheduler class="org.apache.lucene.index.ConcurrentMergeScheduler"/>+
      </indexConfig>

      <autoCommit>
      <maxTime>${solr.autoCommit.maxTime:15000}</maxTime>
      <openSearcher>false</openSearcher>
      </autoCommit>
      <autoSoftCommit>
      <maxTime>${solr.autoSoftCommit.maxTime:-1}</maxTime>
      </autoSoftCommit>

      We are indexing with the following option
      { :batch_commit => false,:batch_size=>20000 }

      We have set autocommit false in our model. So whenever a new record is inserted it is not automatically added to the solr index. But when a record is updated we manually call the Sunspot.index! method for that particular product data. Everyday we are inserting around .2 millions of records. We have target of 50 million products.

      Is there any way, that we can add to index only the new records or updated records?

      Can we increase the indexing speed by changing any of the current configurations?

      If we add new products to the solr through ruby code by using loop, it fails miserably as it takes too much time.

      Please help to find the best way to improve the indexing speed of solr.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              KrishnaSahoo Krishna Sahoo
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: