Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-11010

OutOfMemoryError in tests when using HDFS BlockCache

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 7.0, 8.0
    • 8.1, 9.0
    • Hadoop Integration, hdfs, Tests
    • None

    Description

      Spin-off from SOLR-10878: the newly added MoveReplicaHDFSTest fails on jenkins (but rarely locally) with the following stacktrace:

         [junit4]   2> 13619 ERROR (qtp1885193567-48) [n:127.0.0.1:50324_solr c:movereplicatest_coll s:shard2 r:core_node4 x:movereplicatest_coll_shard2_replica_n2] o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: Error CREATEing SolrCore 'movereplicatest_coll_shard2_replica_n2': Unable to create core [movereplicatest_coll_shard2_replica_n2] Caused by: Direct buffer memory
   [junit4]   2> 	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:938)
   [junit4]   2> 	at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$164(CoreAdminOperation.java:91)
   [junit4]   2> 	at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:384)
   [junit4]   2> 	at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:389)
   [junit4]   2> 	at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:174)
   [junit4]   2> 	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)
   [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:745)
   [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:726)
   [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:507)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:378)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
   [junit4]   2> 	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
   [junit4]   2> 	at org.eclipse.jetty.server.Server.handle(Server.java:534)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
   [junit4]   2> 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
   [junit4]   2> 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
   [junit4]   2> 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> Caused by: org.apache.solr.common.SolrException: Unable to create core [movereplicatest_coll_shard2_replica_n2]
   [junit4]   2> 	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:985)
   [junit4]   2> 	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:904)
   [junit4]   2> 	... 34 more
   [junit4]   2> Caused by: org.apache.solr.common.SolrException: The max direct memory is likely too low.  Either increase it (by adding -XX:MaxDirectMemorySize=<size>g -XX:+UseLargePages to your containers startup args) or disable direct allocation using solr.hdfs.blockcache.direct.memory.allocation=false in solrconfig.xml. If you are putting the block cache on the heap, your java heap size might not be large enough. Failed allocating ~134.217728 MB.
   [junit4]   2> 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:988)
   [junit4]   2> 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:843)
   [junit4]   2> 	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:969)
   [junit4]   2> 	... 35 more
   [junit4]   2> Caused by: java.lang.RuntimeException: The max direct memory is likely too low.  Either increase it (by adding -XX:MaxDirectMemorySize=<size>g -XX:+UseLargePages to your containers startup args) or disable direct allocation using solr.hdfs.blockcache.direct.memory.allocation=false in solrconfig.xml. If you are putting the block cache on the heap, your java heap size might not be large enough. Failed allocating ~134.217728 MB.
   [junit4]   2> 	at org.apache.solr.core.HdfsDirectoryFactory.createBlockCache(HdfsDirectoryFactory.java:311)
   [junit4]   2> 	at org.apache.solr.core.HdfsDirectoryFactory.getBlockDirectoryCache(HdfsDirectoryFactory.java:287)
   [junit4]   2> 	at org.apache.solr.core.HdfsDirectoryFactory.create(HdfsDirectoryFactory.java:227)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)
   [junit4]   2> 	at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:92)
   [junit4]   2> 	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:741)
   [junit4]   2> 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:934)
   [junit4]   2> 	... 37 more
   [junit4]   2> Caused by: java.lang.OutOfMemoryError: Direct buffer memory
   [junit4]   2> 	at java.nio.Bits.reserveMemory(Bits.java:658)
   [junit4]   2> 	at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
   [junit4]   2> 	at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
   [junit4]   2> 	at org.apache.solr.store.blockcache.BlockCache.<init>(BlockCache.java:71)
   [junit4]   2> 	at org.apache.solr.core.HdfsDirectoryFactory.createBlockCache(HdfsDirectoryFactory.java:309)
   [junit4]   2> 	... 43 more

      Attachments

        Issue Links

          is related to

          Test - A new unit, integration or system test. SOLR-13330 Improve HDFS tests

          • Major - Major loss of function.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. SOLR-12080 Frequent failures of MoveReplicaHDFSTest.testFailedMove

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. SOLR-13074 MoveReplicaHDFSTest leaks threads, falls into an endless loop, logging like crazy

          • Major - Major loss of function.
          • Closed

          Activity

            People

              krisden Kevin Risden
              ab Andrzej Bialecki
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: