Solr
  1. Solr
  2. SOLR-4117

IO error while trying to get the size of the Directory

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 6.0
    • Fix Version/s: 4.1, 6.0
    • Component/s: SolrCloud
    • Labels:
      None
    • Environment:

      5.0.0.2012.11.28.10.42.06
      Debian Squeeze, Tomcat 6, Sun Java 6, 10 nodes, 10 shards, rep. factor 2.

      Description

      With SOLR-4032 fixed we see other issues when randomly taking down nodes (nicely via tomcat restart) while indexing a few million web pages from Hadoop. We do make sure that at least one node is up for a shard but due to recovery issues it may not be live.

      One node seems to work but generates IO errors in the log and ZookeeperExeption in the GUI. In the GUI we only see:

      
      SolrCore Initialization Failures
      
          openindex_f: org.apache.solr.common.cloud.ZooKeeperException:org.apache.solr.common.cloud.ZooKeeperException: 
      
      Please check your logs for more information
      

      and in the log we only see the following exception:

      2012-11-28 11:47:26,652 ERROR [solr.handler.ReplicationHandler] - [http-8080-exec-28] - : IO error while trying to get the size of the Directory:org.apache.lucene.store.NoSuchDirectoryException: directory '/opt/solr/cores/shard_f/data/index' does not exist
              at org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:217)
              at org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:240)
              at org.apache.lucene.store.NRTCachingDirectory.listAll(NRTCachingDirectory.java:132)
              at org.apache.solr.core.DirectoryFactory.sizeOfDirectory(DirectoryFactory.java:146)
              at org.apache.solr.handler.ReplicationHandler.getIndexSize(ReplicationHandler.java:472)
              at org.apache.solr.handler.ReplicationHandler.getReplicationDetails(ReplicationHandler.java:568)
              at org.apache.solr.handler.ReplicationHandler.handleRequestBody(ReplicationHandler.java:213)
              at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:144)
              at org.apache.solr.core.RequestHandlers$LazyRequestHandlerWrapper.handleRequest(RequestHandlers.java:240)
              at org.apache.solr.core.SolrCore.execute(SolrCore.java:1830)
              at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:476)
              at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:276)
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
              at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
              at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
              at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
              at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
              at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
              at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
              at org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889)
              at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:744)
              at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2274)
              at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
              at java.lang.Thread.run(Thread.java:662)
      

        Issue Links

          Activity

          Hide
          Markus Jelsma added a comment - - edited

          This issue is the same as reported in SOLR-4032. It does not resolve itself, as it did before in SOLR-4032, when reloading a core or restarting the servlet container. The Zookeeper exception in the GUI is gone after restart so it's likely not related.

          edit: the index.properties file in both cores point to the correct index.LARGE_NUMBER directory but NRTDir tries ./data/index regardless.

          Show
          Markus Jelsma added a comment - - edited This issue is the same as reported in SOLR-4032 . It does not resolve itself, as it did before in SOLR-4032 , when reloading a core or restarting the servlet container. The Zookeeper exception in the GUI is gone after restart so it's likely not related. edit: the index.properties file in both cores point to the correct index.LARGE_NUMBER directory but NRTDir tries ./data/index regardless.
          Hide
          Markus Jelsma added a comment -

          I have another node now logging the same exception for a core that has 0 docs which is not the leader but clusterstate says the node is active and does not attempt recovery. To my surprise it has two index.NUMBER directories of different sizes and index.properties points to the largest directory.

          The node won't come back up properly. Search and indexing works but accessing the GUI is impossible:

          2012-11-28 14:50:00,026 ERROR [solr.servlet.SolrDispatchFilter] - [http-8080-exec-6] - : null:org.apache.solr.common.SolrException: Error handling 'status' action 
                  at org.apache.solr.handler.admin.CoreAdminHandler.handleStatusAction(CoreAdminHandler.java:724)
                  at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:157)
                  at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:144)
                  at org.apache.solr.servlet.SolrDispatchFilter.handleAdminRequest(SolrDispatchFilter.java:372)
                  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:181)
                  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
                  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
                  at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
                  at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
                  at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
                  at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
                  at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
                  at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
                  at org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889)
                  at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:744)
                  at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2274)
                  at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
                  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
                  at java.lang.Thread.run(Thread.java:662)
          Caused by: org.apache.solr.common.SolrException: java.util.concurrent.RejectedExecutionException
                  at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1674)
                  at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1330)
                  at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1265)
                  at org.apache.solr.handler.admin.CoreAdminHandler.getCoreStatus(CoreAdminHandler.java:996)
                  at org.apache.solr.handler.admin.CoreAdminHandler.handleStatusAction(CoreAdminHandler.java:710)
                  ... 18 more
          Caused by: java.util.concurrent.RejectedExecutionException
                  at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1768)
                  at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
                  at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
                  at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:92)
                  at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:603)
                  at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1605)
                  ... 22 more
          
          Show
          Markus Jelsma added a comment - I have another node now logging the same exception for a core that has 0 docs which is not the leader but clusterstate says the node is active and does not attempt recovery. To my surprise it has two index.NUMBER directories of different sizes and index.properties points to the largest directory. The node won't come back up properly. Search and indexing works but accessing the GUI is impossible: 2012-11-28 14:50:00,026 ERROR [solr.servlet.SolrDispatchFilter] - [http-8080-exec-6] - : null :org.apache.solr.common.SolrException: Error handling 'status' action at org.apache.solr.handler.admin.CoreAdminHandler.handleStatusAction(CoreAdminHandler.java:724) at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:157) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:144) at org.apache.solr.servlet.SolrDispatchFilter.handleAdminRequest(SolrDispatchFilter.java:372) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:181) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293) at org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889) at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:744) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2274) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang. Thread .run( Thread .java:662) Caused by: org.apache.solr.common.SolrException: java.util.concurrent.RejectedExecutionException at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1674) at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1330) at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1265) at org.apache.solr.handler.admin.CoreAdminHandler.getCoreStatus(CoreAdminHandler.java:996) at org.apache.solr.handler.admin.CoreAdminHandler.handleStatusAction(CoreAdminHandler.java:710) ... 18 more Caused by: java.util.concurrent.RejectedExecutionException at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1768) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:92) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:603) at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1605) ... 22 more
          Hide
          Eks Dev added a comment - - edited

          fwiw, we think we observed the following problem in simple master slave setup with NRTCachingDirectory... I am not sure it has something to do with issue, because ewe did not see this exception, anyhow

          on replication, slave gets the index from master and works fine, then on:
          1. graceful restart, the world looks fine
          2. kill -9 or such, solr does not start because an index gets corrupt (should actually not happen)

          We speculate that solr now does replication directly to Directory implementation and does not ensure that replicated files get fsck-ed completely after replication. As far as I remember, replication was going to /temp (disk) and than moving files if all went ok. Working under assumption that everything is already persisted. Maybe this invariant does not hold any more and some explicit fsck is needed for caching directories?

          I might be completely wrong, we just observed symptoms in not really debug-friendly environment

          Here Exception after "hard" restart:

          Caused by: org.apache.solr.common.SolrException: Error opening new searcher
          at org.apache.solr.core.SolrCore.<init>(SolrCore.java:804)
          at org.apache.solr.core.SolrCore.<init>(SolrCore.java:618)
          at org.apache.solr.core.CoreContainer.createFromLocal(CoreContainer.java:973)
          at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1003)
          ... 10 more
          Caused by: org.apache.solr.common.SolrException: Error opening new searcher
          at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1441)
          at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1553)
          at org.apache.solr.core.SolrCore.<init>(SolrCore.java:779)
          ... 13 more
          Caused by: java.io.FileNotFoundException: ...\core0\data\index\segments_1 (The system cannot find the file specified)
          at java.io.RandomAccessFile.open(Native Method)
          at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
          at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:222)
          at org.apache.lucene.store.NRTCachingDirectory.openInput(NRTCachingDirectory.java:232)
          at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:281)
          at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:56)
          at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:668)
          at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:52)
          at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:87)
          at org.apache.solr.core.StandardIndexReaderFactory.newReader(StandardIndexReaderFactory.java:34)
          at org.apache.solr.search.SolrIndexSearcher.<init>(SolrIndexSearcher.java:120)
          at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1417)
          ....

          Show
          Eks Dev added a comment - - edited fwiw, we think we observed the following problem in simple master slave setup with NRTCachingDirectory... I am not sure it has something to do with issue, because ewe did not see this exception, anyhow on replication, slave gets the index from master and works fine, then on: 1. graceful restart, the world looks fine 2. kill -9 or such, solr does not start because an index gets corrupt (should actually not happen) We speculate that solr now does replication directly to Directory implementation and does not ensure that replicated files get fsck-ed completely after replication. As far as I remember, replication was going to /temp (disk) and than moving files if all went ok. Working under assumption that everything is already persisted. Maybe this invariant does not hold any more and some explicit fsck is needed for caching directories? I might be completely wrong, we just observed symptoms in not really debug-friendly environment Here Exception after "hard" restart: Caused by: org.apache.solr.common.SolrException: Error opening new searcher at org.apache.solr.core.SolrCore.<init>(SolrCore.java:804) at org.apache.solr.core.SolrCore.<init>(SolrCore.java:618) at org.apache.solr.core.CoreContainer.createFromLocal(CoreContainer.java:973) at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1003) ... 10 more Caused by: org.apache.solr.common.SolrException: Error opening new searcher at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1441) at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1553) at org.apache.solr.core.SolrCore.<init>(SolrCore.java:779) ... 13 more Caused by: java.io.FileNotFoundException: ...\core0\data\index\segments_1 (The system cannot find the file specified) at java.io.RandomAccessFile.open(Native Method) at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233) at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:222) at org.apache.lucene.store.NRTCachingDirectory.openInput(NRTCachingDirectory.java:232) at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:281) at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:56) at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:668) at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:52) at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:87) at org.apache.solr.core.StandardIndexReaderFactory.newReader(StandardIndexReaderFactory.java:34) at org.apache.solr.search.SolrIndexSearcher.<init>(SolrIndexSearcher.java:120) at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1417) ....
          Hide
          Mark Miller added a comment -

          Do you mean fsync rather than fsck (isnt that a file system check?)

          That did change in that we are now using the Directory's sync method - but it should still work the same as before...

          2 should not happen though - so we should dig in. I'm guessing it's not related to this issue, but we will see.

          Show
          Mark Miller added a comment - Do you mean fsync rather than fsck (isnt that a file system check?) That did change in that we are now using the Directory's sync method - but it should still work the same as before... 2 should not happen though - so we should dig in. I'm guessing it's not related to this issue, but we will see.
          Hide
          Eks Dev added a comment -

          fsync of course, fsck was intended for my terminal window

          Show
          Eks Dev added a comment - fsync of course, fsck was intended for my terminal window
          Hide
          Mark Miller added a comment -

          Markus, I'm about to commit a fix to this issue - but I doubt it's the same as the issue you then mention in a comment.

          Show
          Mark Miller added a comment - Markus, I'm about to commit a fix to this issue - but I doubt it's the same as the issue you then mention in a comment.
          Hide
          Commit Tag Bot added a comment -

          [trunk commit] Mark Robert Miller
          http://svn.apache.org/viewvc?view=revision&revision=1414773

          SOLR-4117: Retrieving the size of the index may use the wrong index dir if you are replicating.

          Show
          Commit Tag Bot added a comment - [trunk commit] Mark Robert Miller http://svn.apache.org/viewvc?view=revision&revision=1414773 SOLR-4117 : Retrieving the size of the index may use the wrong index dir if you are replicating.
          Hide
          Commit Tag Bot added a comment -

          [branch_4x commit] Mark Robert Miller
          http://svn.apache.org/viewvc?view=revision&revision=1414774

          SOLR-4117: Retrieving the size of the index may use the wrong index dir if you are replicating.

          Show
          Commit Tag Bot added a comment - [branch_4x commit] Mark Robert Miller http://svn.apache.org/viewvc?view=revision&revision=1414774 SOLR-4117 : Retrieving the size of the index may use the wrong index dir if you are replicating.
          Hide
          Markus Jelsma added a comment -

          Likely indeed. I'll check on this issue tomorrow and try to reproduce the other one, will open new issue if i can.

          Thanks

          Show
          Markus Jelsma added a comment - Likely indeed. I'll check on this issue tomorrow and try to reproduce the other one, will open new issue if i can. Thanks
          Hide
          Markus Jelsma added a comment -

          For some reason we're seeing this again on a node with today's trunk check out:

          2012-12-03 13:46:46,300 ERROR [handler.admin.CoreAdminHandler] - [http-8080-exec-10] - : IO error while trying to get the size of the Directory:org.apache.lucene.store.NoSuchDirectoryException: directory '/opt/solr/cores/core_b/data/index' does not exist
                  at org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:217)
                  at org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:240)
                  at org.apache.lucene.store.NRTCachingDirectory.listAll(NRTCachingDirectory.java:132)
                  at org.apache.solr.core.DirectoryFactory.sizeOfDirectory(DirectoryFactory.java:146)
                  at org.apache.solr.handler.admin.CoreAdminHandler.getIndexSize(CoreAdminHandler.java:1020)
                  at org.apache.solr.handler.admin.CoreAdminHandler.getCoreStatus(CoreAdminHandler.java:1000)
                  at org.apache.solr.handler.admin.CoreAdminHandler.handleStatusAction(CoreAdminHandler.java:711)
                  at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:158)
                  at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:144)
                  at org.apache.solr.servlet.SolrDispatchFilter.handleAdminRequest(SolrDispatchFilter.java:372)
                  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:181)
                  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
                  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
                  at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
                  at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
                  at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
                  at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
                  at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
                  at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
                  at org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889)
                  at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:744)
                  at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2274)
                  at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
                  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
                  at java.lang.Thread.run(Thread.java:662)
          
          Show
          Markus Jelsma added a comment - For some reason we're seeing this again on a node with today's trunk check out: 2012-12-03 13:46:46,300 ERROR [handler.admin.CoreAdminHandler] - [http-8080-exec-10] - : IO error while trying to get the size of the Directory:org.apache.lucene.store.NoSuchDirectoryException: directory '/opt/solr/cores/core_b/data/index' does not exist at org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:217) at org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:240) at org.apache.lucene.store.NRTCachingDirectory.listAll(NRTCachingDirectory.java:132) at org.apache.solr.core.DirectoryFactory.sizeOfDirectory(DirectoryFactory.java:146) at org.apache.solr.handler.admin.CoreAdminHandler.getIndexSize(CoreAdminHandler.java:1020) at org.apache.solr.handler.admin.CoreAdminHandler.getCoreStatus(CoreAdminHandler.java:1000) at org.apache.solr.handler.admin.CoreAdminHandler.handleStatusAction(CoreAdminHandler.java:711) at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:158) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:144) at org.apache.solr.servlet.SolrDispatchFilter.handleAdminRequest(SolrDispatchFilter.java:372) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:181) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293) at org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889) at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:744) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2274) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang. Thread .run( Thread .java:662)
          Hide
          Mark Miller added a comment -

          Thanks Markus - another race around updating to the new index and looking for the size of the index. I'll fix this.

          Show
          Mark Miller added a comment - Thanks Markus - another race around updating to the new index and looking for the size of the index. I'll fix this.
          Hide
          Mark Miller added a comment -

          Here is a patch that does 2 things:

          If we find the directory and see a file listed, but then get a file not found trying to access it (it was removed out from under us), just return a 0 size

          Also, if we can't find the directory at all, try using the newIndexDir.

          Show
          Mark Miller added a comment - Here is a patch that does 2 things: If we find the directory and see a file listed, but then get a file not found trying to access it (it was removed out from under us), just return a 0 size Also, if we can't find the directory at all, try using the newIndexDir.
          Hide
          Commit Tag Bot added a comment -

          [trunk commit] Mark Robert Miller
          http://svn.apache.org/viewvc?view=revision&revision=1418725

          SOLR-4117: harden size of directory code

          Show
          Commit Tag Bot added a comment - [trunk commit] Mark Robert Miller http://svn.apache.org/viewvc?view=revision&revision=1418725 SOLR-4117 : harden size of directory code
          Hide
          Shawn Heisey added a comment -

          Mark, it seems like this is also likely to resolve SOLR-4135. Can you look into that?

          I was seeing another issue in my log that Yonik thought might be tied to the disappearing file issue. If I continue to get that after this patch, I will open another issue.

          Show
          Shawn Heisey added a comment - Mark, it seems like this is also likely to resolve SOLR-4135 . Can you look into that? I was seeing another issue in my log that Yonik thought might be tied to the disappearing file issue. If I continue to get that after this patch, I will open another issue.
          Hide
          Hoss Man added a comment -

          I think all of the changes in r1418725 got merged to 4x as part of r1420992...

          http://svn.apache.org/viewvc?view=revision&revision=1420992

          (or in any event: attempting to merge r1418725 into 4x as of r1421071 is a No-Op)

          Show
          Hoss Man added a comment - I think all of the changes in r1418725 got merged to 4x as part of r1420992... http://svn.apache.org/viewvc?view=revision&revision=1420992 (or in any event: attempting to merge r1418725 into 4x as of r1421071 is a No-Op)
          Hide
          Commit Tag Bot added a comment -

          [branch_4x commit] Mark Robert Miller
          http://svn.apache.org/viewvc?view=revision&revision=1414774

          SOLR-4117: Retrieving the size of the index may use the wrong index dir if you are replicating.

          Show
          Commit Tag Bot added a comment - [branch_4x commit] Mark Robert Miller http://svn.apache.org/viewvc?view=revision&revision=1414774 SOLR-4117 : Retrieving the size of the index may use the wrong index dir if you are replicating.

            People

            • Assignee:
              Mark Miller
              Reporter:
              Markus Jelsma
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development