Solr
  1. Solr
  2. SOLR-5214

collections?action=SPLITSHARD running out of heap space due to merge

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 4.3
    • Fix Version/s: 4.7, 6.0
    • Component/s: update
    • Labels:
      None

      Description

      The problem we saw was that splitting a shard with many segments and documents
      failed by running out of heap space.

      Increasing heap space so that all existing segments could be merged into one
      overall segment does not seem practical. Running the split without segment
      merging worked.

      Could split always run without merging, or merge=true/false be an optional parameter for the SPLITSHARD action?

      1. SOLR-5214.patch
        2 kB
        Christine Poerschke

        Activity

        Hide
        Christine Poerschke added a comment -

        Attaching patch against trunk, to not merge when splitting (i.e. no merge=true/false parameter as yet).

        Show
        Christine Poerschke added a comment - Attaching patch against trunk, to not merge when splitting (i.e. no merge=true/false parameter as yet).
        Hide
        Christine Poerschke added a comment -

        Correcting subReaders.length vs. leaves.size() typo in my original patch.

        Show
        Christine Poerschke added a comment - Correcting subReaders.length vs. leaves.size() typo in my original patch.
        Hide
        Shalin Shekhar Mangar added a comment -

        Thanks Christine. Do you have the OutOfMemoryError stack trace?

        Show
        Shalin Shekhar Mangar added a comment - Thanks Christine. Do you have the OutOfMemoryError stack trace?
        Hide
        Christine Poerschke added a comment -

        Hello. Here's one of the stack traces. And in case it's useful context, during the shard split indexing into the cloud had been stopped but periodic admin/luke and admin/mbeans cat=CACHE stats requests were happening.

        2013-09-03 07:27:51,947 ERROR [qtp1533478516-49] o.a.s.s.SolrDispatchFilter [SolrException.java:119] null:java.lang.OutOfMemoryError: Java heap space
                at java.lang.StringCoding.decode(StringCoding.java:215)
                at java.lang.String.<init>(String.java:453)
                at java.lang.String.<init>(String.java:505)
                at org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.readField(CompressingStoredFieldsReader.java:154)
                at org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.visitDocument(CompressingStoredFieldsReader.java:272)
                at org.apache.lucene.index.SegmentReader.document(SegmentReader.java:133)
                at org.apache.lucene.index.FilterAtomicReader.document(FilterAtomicReader.java:365)
                at org.apache.lucene.index.IndexReader.document(IndexReader.java:436)
                at org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.merge(CompressingStoredFieldsWriter.java:332)
                at org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:298)
                at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:86)
                at org.apache.lucene.index.IndexWriter.addIndexes(IndexWriter.java:2448)
                at org.apache.solr.update.SolrIndexSplitter.split(SolrIndexSplitter.java:118)
                at org.apache.solr.update.DirectUpdateHandler2.split(DirectUpdateHandler2.java:749)
                at org.apache.solr.handler.admin.CoreAdminHandler.handleSplitAction(CoreAdminHandler.java:282)
                at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:185)
                at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
                at org.apache.solr.servlet.SolrDispatchFilter.handleAdminRequest(SolrDispatchFilter.java:608)
                at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:206)
                at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:155)
                at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
                at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
                at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
                at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
                at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
                at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
                at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
                at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
                at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
                at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
                at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
                at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
        
        Show
        Christine Poerschke added a comment - Hello. Here's one of the stack traces. And in case it's useful context, during the shard split indexing into the cloud had been stopped but periodic admin/luke and admin/mbeans cat=CACHE stats requests were happening. 2013-09-03 07:27:51,947 ERROR [qtp1533478516-49] o.a.s.s.SolrDispatchFilter [SolrException.java:119] null:java.lang.OutOfMemoryError: Java heap space at java.lang.StringCoding.decode(StringCoding.java:215) at java.lang.String.<init>(String.java:453) at java.lang.String.<init>(String.java:505) at org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.readField(CompressingStoredFieldsReader.java:154) at org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.visitDocument(CompressingStoredFieldsReader.java:272) at org.apache.lucene.index.SegmentReader.document(SegmentReader.java:133) at org.apache.lucene.index.FilterAtomicReader.document(FilterAtomicReader.java:365) at org.apache.lucene.index.IndexReader.document(IndexReader.java:436) at org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.merge(CompressingStoredFieldsWriter.java:332) at org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:298) at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:86) at org.apache.lucene.index.IndexWriter.addIndexes(IndexWriter.java:2448) at org.apache.solr.update.SolrIndexSplitter.split(SolrIndexSplitter.java:118) at org.apache.solr.update.DirectUpdateHandler2.split(DirectUpdateHandler2.java:749) at org.apache.solr.handler.admin.CoreAdminHandler.handleSplitAction(CoreAdminHandler.java:282) at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:185) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135) at org.apache.solr.servlet.SolrDispatchFilter.handleAdminRequest(SolrDispatchFilter.java:608) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:206) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:155) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
        Hide
        ASF subversion and git services added a comment -

        Commit 1553644 from shalin@apache.org in branch 'dev/trunk'
        [ https://svn.apache.org/r1553644 ]

        SOLR-5214: Reduce memory usage for shard splitting by merging segments one at a time

        Show
        ASF subversion and git services added a comment - Commit 1553644 from shalin@apache.org in branch 'dev/trunk' [ https://svn.apache.org/r1553644 ] SOLR-5214 : Reduce memory usage for shard splitting by merging segments one at a time
        Hide
        ASF subversion and git services added a comment -

        Commit 1553645 from shalin@apache.org in branch 'dev/branches/branch_4x'
        [ https://svn.apache.org/r1553645 ]

        SOLR-5214: Reduce memory usage for shard splitting by merging segments one at a time

        Show
        ASF subversion and git services added a comment - Commit 1553645 from shalin@apache.org in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1553645 ] SOLR-5214 : Reduce memory usage for shard splitting by merging segments one at a time
        Hide
        Shalin Shekhar Mangar added a comment -

        Thanks Christine!

        Show
        Shalin Shekhar Mangar added a comment - Thanks Christine!

          People

          • Assignee:
            Shalin Shekhar Mangar
            Reporter:
            Christine Poerschke
          • Votes:
            1 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development