[SOLR-4117] IO error while trying to get the size of the Directory

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 6.0
• Fix Version/s: 4.1, 6.0
• Component/s: SolrCloud

Environment

5.0.0.2012.11.28.10.42.06
Debian Squeeze, Tomcat 6, Sun Java 6, 10 nodes, 10 shards, rep. factor 2.

• Assignee: Mark Miller
• Reporter: Markus Jelsma
• Created: 28/Nov/12 11:49
• Updated: 09/May/16 18:51
• Resolved: 10/Dec/12 23:56

Description

With SOLR-4032 fixed we see other issues when randomly taking down nodes (nicely via tomcat restart) while indexing a few million web pages from Hadoop. We do make sure that at least one node is up for a shard but due to recovery issues it may not be live.

One node seems to work but generates IO errors in the log and ZookeeperExeption in the GUI. In the GUI we only see:

SolrCore Initialization Failures

    openindex_f: org.apache.solr.common.cloud.ZooKeeperException:org.apache.solr.common.cloud.ZooKeeperException: 

Please check your logs for more information

and in the log we only see the following exception:

2012-11-28 11:47:26,652 ERROR [solr.handler.ReplicationHandler] - [http-8080-exec-28] - : IO error while trying to get the size of the Directory:org.apache.lucene.store.NoSuchDirectoryException: directory '/opt/solr/cores/shard_f/data/index' does not exist
        at org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:217)
        at org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:240)
        at org.apache.lucene.store.NRTCachingDirectory.listAll(NRTCachingDirectory.java:132)
        at org.apache.solr.core.DirectoryFactory.sizeOfDirectory(DirectoryFactory.java:146)
        at org.apache.solr.handler.ReplicationHandler.getIndexSize(ReplicationHandler.java:472)
        at org.apache.solr.handler.ReplicationHandler.getReplicationDetails(ReplicationHandler.java:568)
        at org.apache.solr.handler.ReplicationHandler.handleRequestBody(ReplicationHandler.java:213)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:144)
        at org.apache.solr.core.RequestHandlers$LazyRequestHandlerWrapper.handleRequest(RequestHandlers.java:240)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:1830)
        at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:476)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:276)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
        at org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889)
        at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:744)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2274)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)

Issue Links

• contains: SOLR-4135 java.lang.IllegalArgumentException when getting index size in replication handler
• is related to: SOLR-4116 Log Replay [recoveryExecutor-8-thread-1] - : java.io.EOFException
• relates to: SOLR-4032 Files larger than an internal buffer size fail to replicate
• relates to: SOLR-4033 No lockType configured for NRTCachingDirectory
• relates to: SOLR-4135 java.lang.IllegalArgumentException when getting index size in replication handler