Solr
  1. Solr
  2. SOLR-1150

OutofMemoryError on enabling highlighting

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.4
    • Fix Version/s: 1.4
    • Component/s: highlighter
    • Labels:
      None

      Description

      Please refer to following mail thread

      http://markmail.org/message/5nhkm5h3ongqlput

      I am testing with 2MB document size and just 500 documents. Indexing is working fine even with 128MB heap size. But on searching Solr throws OOM error. This issue is observed only when we enable highlighting. While indexing I am storing 1 MB text. While searching Solr reads all the 500 documents in the memory. It also reads the complete 1 MB stored field in the memory for all 500 documents. Due to this 500 docs * 1 MB * 2 (2 bytes per char) = 1000 MB memory is required for searching.

      This memory usage can be reduced by reading one document at a time.

      1. SOLR-1150.patch
        2 kB
        Mark Miller
      2. SOLR-1150.patch
        2 kB
        Siddharth Gargate

        Activity

        Hide
        Siddharth Gargate added a comment -

        I have attached the patch file with previous suggested code changes..

        Show
        Siddharth Gargate added a comment - I have attached the patch file with previous suggested code changes..
        Hide
        Mark Miller added a comment -

        Patch looks right - my only concern would be a performance impact, but that does look unlikely. There is likely to be some benefit in loading them all at once, but I can't imagine one at a time is much of a loss.

        Show
        Mark Miller added a comment - Patch looks right - my only concern would be a performance impact, but that does look unlikely. There is likely to be some benefit in loading them all at once, but I can't imagine one at a time is much of a loss.
        Hide
        Siddharth Gargate added a comment -

        Thanks Mark.
        SolrIndexSearcher.readDocs method internally reads one doc at a time. So there shouldn't be any performance loss.

        Show
        Siddharth Gargate added a comment - Thanks Mark. SolrIndexSearcher.readDocs method internally reads one doc at a time. So there shouldn't be any performance loss.
        Hide
        Mark Miller added a comment -

        Odd - change looks like it wouldnt affect this, but somehow the highlighter test fails as it attempts to access a deleted doc. Not quite sure what is up yet.

        Show
        Mark Miller added a comment - Odd - change looks like it wouldnt affect this, but somehow the highlighter test fails as it attempts to access a deleted doc. Not quite sure what is up yet.
        Hide
        Mark Miller added a comment -

        There is a problem with distrib and highlighting as well.

        Show
        Mark Miller added a comment - There is a problem with distrib and highlighting as well.
        Hide
        Yonik Seeley added a comment -

        It's trying to read the loop iterator (i.e. 0-9)

        Show
        Yonik Seeley added a comment - It's trying to read the loop iterator (i.e. 0-9)
        Hide
        Mark Miller added a comment -

        ah, thanks for the spot yonik.

        Ill switch it to use doc ids instead and see how things go

        Show
        Mark Miller added a comment - ah, thanks for the spot yonik. Ill switch it to use doc ids instead and see how things go
        Hide
        Mark Miller added a comment -

        Thanks Siddharth!

        Show
        Mark Miller added a comment - Thanks Siddharth!
        Hide
        Grant Ingersoll added a comment -

        Bulk close for Solr 1.4

        Show
        Grant Ingersoll added a comment - Bulk close for Solr 1.4

          People

          • Assignee:
            Mark Miller
            Reporter:
            Siddharth Gargate
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development