Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-11010

OutOfMemoryError in tests when using HDFS BlockCache

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 7.0, 8.0
    • 8.1, 9.0
    • Hadoop Integration, hdfs, Tests
    • None

    Description

      Spin-off from SOLR-10878: the newly added MoveReplicaHDFSTest fails on jenkins (but rarely locally) with the following stacktrace:

         [junit4]   2> 13619 ERROR (qtp1885193567-48) [n:127.0.0.1:50324_solr c:movereplicatest_coll s:shard2 r:core_node4 x:movereplicatest_coll_shard2_replica_n2] o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: Error CREATEing SolrCore 'movereplicatest_coll_shard2_replica_n2': Unable to create core [movereplicatest_coll_shard2_replica_n2] Caused by: Direct buffer memory
         [junit4]   2> 	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:938)
         [junit4]   2> 	at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$164(CoreAdminOperation.java:91)
         [junit4]   2> 	at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:384)
         [junit4]   2> 	at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:389)
         [junit4]   2> 	at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:174)
         [junit4]   2> 	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)
         [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:745)
         [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:726)
         [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:507)
         [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:378)
         [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322)
         [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
         [junit4]   2> 	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
         [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
         [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
         [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
         [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
         [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
         [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
         [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
         [junit4]   2> 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
         [junit4]   2> 	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)
         [junit4]   2> 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
         [junit4]   2> 	at org.eclipse.jetty.server.Server.handle(Server.java:534)
         [junit4]   2> 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
         [junit4]   2> 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
         [junit4]   2> 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
         [junit4]   2> 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
         [junit4]   2> 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
         [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
         [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
         [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
         [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
         [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
         [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
         [junit4]   2> Caused by: org.apache.solr.common.SolrException: Unable to create core [movereplicatest_coll_shard2_replica_n2]
         [junit4]   2> 	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:985)
         [junit4]   2> 	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:904)
         [junit4]   2> 	... 34 more
         [junit4]   2> Caused by: org.apache.solr.common.SolrException: The max direct memory is likely too low.  Either increase it (by adding -XX:MaxDirectMemorySize=<size>g -XX:+UseLargePages to your containers startup args) or disable direct allocation using solr.hdfs.blockcache.direct.memory.allocation=false in solrconfig.xml. If you are putting the block cache on the heap, your java heap size might not be large enough. Failed allocating ~134.217728 MB.
         [junit4]   2> 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:988)
         [junit4]   2> 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:843)
         [junit4]   2> 	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:969)
         [junit4]   2> 	... 35 more
         [junit4]   2> Caused by: java.lang.RuntimeException: The max direct memory is likely too low.  Either increase it (by adding -XX:MaxDirectMemorySize=<size>g -XX:+UseLargePages to your containers startup args) or disable direct allocation using solr.hdfs.blockcache.direct.memory.allocation=false in solrconfig.xml. If you are putting the block cache on the heap, your java heap size might not be large enough. Failed allocating ~134.217728 MB.
         [junit4]   2> 	at org.apache.solr.core.HdfsDirectoryFactory.createBlockCache(HdfsDirectoryFactory.java:311)
         [junit4]   2> 	at org.apache.solr.core.HdfsDirectoryFactory.getBlockDirectoryCache(HdfsDirectoryFactory.java:287)
         [junit4]   2> 	at org.apache.solr.core.HdfsDirectoryFactory.create(HdfsDirectoryFactory.java:227)
         [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)
         [junit4]   2> 	at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:92)
         [junit4]   2> 	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:741)
         [junit4]   2> 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:934)
         [junit4]   2> 	... 37 more
         [junit4]   2> Caused by: java.lang.OutOfMemoryError: Direct buffer memory
         [junit4]   2> 	at java.nio.Bits.reserveMemory(Bits.java:658)
         [junit4]   2> 	at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
         [junit4]   2> 	at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
         [junit4]   2> 	at org.apache.solr.store.blockcache.BlockCache.<init>(BlockCache.java:71)
         [junit4]   2> 	at org.apache.solr.core.HdfsDirectoryFactory.createBlockCache(HdfsDirectoryFactory.java:309)
         [junit4]   2> 	... 43 more
      

      Attachments

        Issue Links

          Activity

            People

              krisden Kevin Risden
              ab Andrzej Bialecki
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: