[SOLR-6480] Too Many Open files trying to ask a replica to recover

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 4.7.2
• Assignee: Timothy Potter
• Reporter: Ugo Matrangolo
• Created: 04/Sep/14 14:56
• Updated: 22/Apr/16 11:47
• Resolved: 22/Apr/16 11:47

Description

After the DistributedUpdateProcessor tries to ask multiple times a replica to recover it eventually starts to fail with the following error:

2014-08-28 22:42:46,285 [updateExecutor-1-thread-2334] ERROR org.apache.solr.update.processor.DistributedUpdateProcessor  - http://10.140.4.246:9765: Could not tell a replica to recover:org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://10.140.4.246:9765
        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:507)
        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:199)
        at org.apache.solr.update.processor.DistributedUpdateProcessor$1.run(DistributedUpdateProcessor.java:685)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.SocketException: Too many open files
        at java.net.Socket.createImpl(Socket.java:397)
        at java.net.Socket.getImpl(Socket.java:460)
        at java.net.Socket.setSoTimeout(Socket.java:1017)
        at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:116)
        at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:178)
        at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:304)
        at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:610)
        at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:445)
        at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:395)
        ... 5 more

Issue Links

• is part of: SOLR-5961 Solr gets crazy on /overseer/queue state change