Solr
  1. Solr
  2. SOLR-2290

the termsInfosDivisor for readers opened by indexWriter should be configurable in Solr

    Details

    • Type: New Feature New Feature
    • Status: Open
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Solr allows users to set the termInfosIndexDivisor used by the indexReader during search time in solrconfig.xml, but not in the indexReader opened by the IndexWriter when indexing/merging.

      When dealing with an index with a large number of unique terms, setting the termInfosIndexDivisor at search time is helpful in reducing memory use. It would also be helpful in reducing memory use during indexing/merging if it was made configurable for indexReaders opened by indexWriter during indexing/merging.

      This thread contains some background:
      http://www.lucidimagination.com/search/document/b5c756a366e1a0d6/memory_use_during_merges_oom

      In the Lucene 3.x branch it looks like this is done in IndexWriterConfig.setReaderTermsIndexDivisor, although there is also this method signature in IndexWriter.java: IndexReader getReader(int termInfosIndexDivisor)

        

        Activity

        Hide
        Jason Rutherglen added a comment -

        Tom, I think this can be generified to use SOLR-1447's property injection into IWC.

        Show
        Jason Rutherglen added a comment - Tom, I think this can be generified to use SOLR-1447 's property injection into IWC.
        Hide
        Tom Burton-West added a comment -

        Thanks Jason,

        I'm still working my way though the Solr codebase and don't yet really understand the relationship between the lucene classes and the solr classes and the Solr config process.

        I'd like to do the property injection. I'm having trouble conceptualizing what the entry would be in the solrconfig.xml file
        and what the name would be in SolrIndexConfig. How do we distinguish between the indexReader termInfoIndexDivisor used during search and the indexReader termInfoIndexDivisor for the indexReader that is used by the indexWriter?

        Would it be something like this?

        <<indexWriter name="IndexWriter" class="org.apache.solr.update.SolrIndexWriter">
        <int name="termInfosIndexDivisor">8</int>
        </indexWriter>

        Maybe you could point me to the classes I should be looking at?

        Tom

        Show
        Tom Burton-West added a comment - Thanks Jason, I'm still working my way though the Solr codebase and don't yet really understand the relationship between the lucene classes and the solr classes and the Solr config process. I'd like to do the property injection. I'm having trouble conceptualizing what the entry would be in the solrconfig.xml file and what the name would be in SolrIndexConfig. How do we distinguish between the indexReader termInfoIndexDivisor used during search and the indexReader termInfoIndexDivisor for the indexReader that is used by the indexWriter? Would it be something like this? <<indexWriter name="IndexWriter" class="org.apache.solr.update.SolrIndexWriter"> <int name="termInfosIndexDivisor">8</int> </indexWriter> Maybe you could point me to the classes I should be looking at? Tom
        Hide
        Jason Rutherglen added a comment -

        I think it'll require creating a new sub-element of mainIndex and indexDefaults called perhaps indexWriterConfig? Because attributes such as unlockOnStartup and reopenReaders cannot be injected in, and we probably don't want to mix injected properties with non-injected properties?

        Show
        Jason Rutherglen added a comment - I think it'll require creating a new sub-element of mainIndex and indexDefaults called perhaps indexWriterConfig? Because attributes such as unlockOnStartup and reopenReaders cannot be injected in, and we probably don't want to mix injected properties with non-injected properties?

          People

          • Assignee:
            Unassigned
            Reporter:
            Tom Burton-West
          • Votes:
            1 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:

              Development