Solr
  1. Solr
  2. SOLR-6161

OutOfMemoryError Not Thrown in sendError

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.9, 6.0
    • Component/s: SolrCloud
    • Labels:
      None

      Description

      We need to handle the OOME thrown here.
      https://github.com/apache/lucene-solr/blob/trunk/solr/core/src/java/org/apache/solr/servlet/SolrDispatchFilter.java#L436

      I experienced this in my solr server logs.
      ERROR org.apache.solr.servlet.SolrDispatchFilter: null:java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap space
      at org.apache.solr.servlet.SolrDispatchFilter.sendError(SolrDispatchFilter.java:775)
      at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:419)
      at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:187)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      at org.apache.solr.servlet.ProxyUserFilter.doFilter(ProxyUserFilter.java:241)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      at org.apache.solr.servlet.SolrHadoopAuthenticationFilter$2.doFilter(SolrHadoopAuthenticationFilter.java:143)
      at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:384)
      at org.apache.solr.servlet.SolrHadoopAuthenticationFilter.doFilter(SolrHadoopAuthenticationFilter.java:148)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      at org.apache.solr.servlet.HostnameFilter.doFilter(HostnameFilter.java:86)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
      at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
      at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
      at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
      at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
      at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861)
      at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:606)
      at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
      at java.lang.Thread.run(Thread.java:744)
      Caused by: java.lang.OutOfMemoryError: Java heap space
      at org.apache.lucene.index.DocTermOrds.uninvert(DocTermOrds.java:279)
      at org.apache.solr.request.UnInvertedField.<init>(UnInvertedField.java:179)
      at org.apache.solr.request.UnInvertedField.getUnInvertedField(UnInvertedField.java:664)
      at org.apache.solr.request.SimpleFacets.getTermCounts(SimpleFacets.java:435)
      at org.apache.solr.handler.component.PivotFacetHelper.doPivots(PivotFacetHelper.java:147)
      at org.apache.solr.handler.component.PivotFacetHelper.process(PivotFacetHelper.java:90)
      at org.apache.solr.handler.component.FacetComponent.process(FacetComponent.java:85)
      at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:208)
      at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
      at org.apache.solr.core.SolrCore.execute(SolrCore.java:1947)
      at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:759)
      at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:398)
      at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:187)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      at org.apache.solr.servlet.ProxyUserFilter.doFilter(ProxyUserFilter.java:241)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      at org.apache.solr.servlet.SolrHadoopAuthenticationFilter$2.doFilter(SolrHadoopAuthenticationFilter.java:143)
      at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:384)
      at org.apache.solr.servlet.SolrHadoopAuthenticationFilter.doFilter(SolrHadoopAuthenticationFilter.java:148)

        Activity

        Hide
        Shalin Shekhar Mangar added a comment -

        Which version of Solr are you running? A bunch of these issues were solved in SOLR-4992 in Solr 4.7

        Show
        Shalin Shekhar Mangar added a comment - Which version of Solr are you running? A bunch of these issues were solved in SOLR-4992 in Solr 4.7
        Hide
        Miklos Christine added a comment -

        I'm using CDH5.0.1, which is based off of Solr 4.4.

        I reviewed the latest patch for SOLR-4992 and didn't see the exceptions being handled explicitly as done in other sections of the code.

        Show
        Miklos Christine added a comment - I'm using CDH5.0.1, which is based off of Solr 4.4. I reviewed the latest patch for SOLR-4992 and didn't see the exceptions being handled explicitly as done in other sections of the code.
        Hide
        Shalin Shekhar Mangar added a comment -

        The OutOfMemoryError was wrapped in a RuntimeException which is why it was not thrown to the JVM. I have reviewed the solr code base, trying to find a place where a Throwable is wrapped in a RuntimeException but I could not find such a place. In any case, it doesn't harm us if we instance check the cause and throw if it was an OOM. I'll fix.

        Show
        Shalin Shekhar Mangar added a comment - The OutOfMemoryError was wrapped in a RuntimeException which is why it was not thrown to the JVM. I have reviewed the solr code base, trying to find a place where a Throwable is wrapped in a RuntimeException but I could not find such a place. In any case, it doesn't harm us if we instance check the cause and throw if it was an OOM. I'll fix.
        Hide
        ASF subversion and git services added a comment -

        Commit 1602590 from shalin@apache.org in branch 'dev/trunk'
        [ https://svn.apache.org/r1602590 ]

        SOLR-6161: SolrDispatchFilter should throw java.lang.Error back even if wrapped in another exception

        Show
        ASF subversion and git services added a comment - Commit 1602590 from shalin@apache.org in branch 'dev/trunk' [ https://svn.apache.org/r1602590 ] SOLR-6161 : SolrDispatchFilter should throw java.lang.Error back even if wrapped in another exception
        Hide
        ASF subversion and git services added a comment -

        Commit 1602591 from shalin@apache.org in branch 'dev/branches/branch_4x'
        [ https://svn.apache.org/r1602591 ]

        SOLR-6161: SolrDispatchFilter should throw java.lang.Error back even if wrapped in another exception

        Show
        ASF subversion and git services added a comment - Commit 1602591 from shalin@apache.org in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1602591 ] SOLR-6161 : SolrDispatchFilter should throw java.lang.Error back even if wrapped in another exception
        Hide
        Shalin Shekhar Mangar added a comment -

        Thanks Miklos!

        Show
        Shalin Shekhar Mangar added a comment - Thanks Miklos!
        Hide
        Mark Miller added a comment -

        Seems like if we are going to do this, we should run up the full cause chain?

        Show
        Mark Miller added a comment - Seems like if we are going to do this, we should run up the full cause chain?
        Hide
        ASF subversion and git services added a comment -

        Commit 1603708 from shalin@apache.org in branch 'dev/trunk'
        [ https://svn.apache.org/r1603708 ]

        SOLR-6161: Walk the entire cause chain looking for an Error

        Show
        ASF subversion and git services added a comment - Commit 1603708 from shalin@apache.org in branch 'dev/trunk' [ https://svn.apache.org/r1603708 ] SOLR-6161 : Walk the entire cause chain looking for an Error
        Hide
        ASF subversion and git services added a comment -

        Commit 1603709 from shalin@apache.org in branch 'dev/branches/branch_4x'
        [ https://svn.apache.org/r1603709 ]

        SOLR-6161: Walk the entire cause chain looking for an Error

        Show
        ASF subversion and git services added a comment - Commit 1603709 from shalin@apache.org in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1603709 ] SOLR-6161 : Walk the entire cause chain looking for an Error
        Hide
        ASF subversion and git services added a comment -

        Commit 1603710 from shalin@apache.org in branch 'dev/branches/lucene_solr_4_9'
        [ https://svn.apache.org/r1603710 ]

        SOLR-6161: Walk the entire cause chain looking for an Error

        Show
        ASF subversion and git services added a comment - Commit 1603710 from shalin@apache.org in branch 'dev/branches/lucene_solr_4_9' [ https://svn.apache.org/r1603710 ] SOLR-6161 : Walk the entire cause chain looking for an Error
        Hide
        Shalin Shekhar Mangar added a comment -

        Thanks for the review Mark. I should have done that in the first go.

        Show
        Shalin Shekhar Mangar added a comment - Thanks for the review Mark. I should have done that in the first go.

          People

          • Assignee:
            Shalin Shekhar Mangar
            Reporter:
            Miklos Christine
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development