Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-1150

OutofMemoryError on enabling highlighting

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.4
    • Fix Version/s: 1.4
    • Component/s: highlighter
    • Labels:
      None

      Description

      Please refer to following mail thread

      http://markmail.org/message/5nhkm5h3ongqlput

      I am testing with 2MB document size and just 500 documents. Indexing is working fine even with 128MB heap size. But on searching Solr throws OOM error. This issue is observed only when we enable highlighting. While indexing I am storing 1 MB text. While searching Solr reads all the 500 documents in the memory. It also reads the complete 1 MB stored field in the memory for all 500 documents. Due to this 500 docs * 1 MB * 2 (2 bytes per char) = 1000 MB memory is required for searching.

      This memory usage can be reduced by reading one document at a time.

      1. SOLR-1150.patch
        2 kB
        Mark Miller
      2. SOLR-1150.patch
        2 kB
        Siddharth Gargate

        Activity

        Hide
        sidgate Siddharth Gargate added a comment -

        I have attached the patch file with previous suggested code changes..

        Show
        sidgate Siddharth Gargate added a comment - I have attached the patch file with previous suggested code changes..
        Hide
        markrmiller@gmail.com Mark Miller added a comment -

        Patch looks right - my only concern would be a performance impact, but that does look unlikely. There is likely to be some benefit in loading them all at once, but I can't imagine one at a time is much of a loss.

        Show
        markrmiller@gmail.com Mark Miller added a comment - Patch looks right - my only concern would be a performance impact, but that does look unlikely. There is likely to be some benefit in loading them all at once, but I can't imagine one at a time is much of a loss.
        Hide
        sidgate Siddharth Gargate added a comment -

        Thanks Mark.
        SolrIndexSearcher.readDocs method internally reads one doc at a time. So there shouldn't be any performance loss.

        Show
        sidgate Siddharth Gargate added a comment - Thanks Mark. SolrIndexSearcher.readDocs method internally reads one doc at a time. So there shouldn't be any performance loss.
        Hide
        markrmiller@gmail.com Mark Miller added a comment -

        Odd - change looks like it wouldnt affect this, but somehow the highlighter test fails as it attempts to access a deleted doc. Not quite sure what is up yet.

        Show
        markrmiller@gmail.com Mark Miller added a comment - Odd - change looks like it wouldnt affect this, but somehow the highlighter test fails as it attempts to access a deleted doc. Not quite sure what is up yet.
        Hide
        markrmiller@gmail.com Mark Miller added a comment -

        There is a problem with distrib and highlighting as well.

        Show
        markrmiller@gmail.com Mark Miller added a comment - There is a problem with distrib and highlighting as well.
        Hide
        yseeley@gmail.com Yonik Seeley added a comment -

        It's trying to read the loop iterator (i.e. 0-9)

        Show
        yseeley@gmail.com Yonik Seeley added a comment - It's trying to read the loop iterator (i.e. 0-9)
        Hide
        markrmiller@gmail.com Mark Miller added a comment -

        ah, thanks for the spot yonik.

        Ill switch it to use doc ids instead and see how things go

        Show
        markrmiller@gmail.com Mark Miller added a comment - ah, thanks for the spot yonik. Ill switch it to use doc ids instead and see how things go
        Hide
        markrmiller@gmail.com Mark Miller added a comment -

        Thanks Siddharth!

        Show
        markrmiller@gmail.com Mark Miller added a comment - Thanks Siddharth!
        Hide
        gsingers Grant Ingersoll added a comment -

        Bulk close for Solr 1.4

        Show
        gsingers Grant Ingersoll added a comment - Bulk close for Solr 1.4

          People

          • Assignee:
            markrmiller@gmail.com Mark Miller
            Reporter:
            sidgate Siddharth Gargate
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development