[SOLR-9305] ForceLeaderTest.testReplicasInLIRNoLeader(): connection refused

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 6.3
• Assignee: Alan Woodward
• Reporter: Steven Rowe
• Created: 13/Jul/16 19:17
• Updated: 08/Jun/19 15:37
• Resolved: 26/Sep/16 22:08

Description

I saw the following while doing other testing on latest master, reproduces for me:

   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=ForceLeaderTest -Dtests.method=testReplicasInLIRNoLeader -Dtests.seed=64E138BA51B16ACE -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=de-LU -Dtests.timezone=America/Indiana/Winamac -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   32.9s | ForceLeaderTest.testReplicasInLIRNoLeader <<<
   [junit4]    > Throwable #1: org.apache.solr.client.solrj.SolrServerException: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://127.0.0.1:59064/forceleader_test_collection_shard1_replica1]
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([64E138BA51B16ACE:82760C7A683393AF]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:753)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1151)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1040)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:976)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.sendDocsWithRetry(AbstractFullDistribZkTestBase.java:753)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.sendDocsWithRetry(AbstractFullDistribZkTestBase.java:741)
   [junit4]    > 	at org.apache.solr.cloud.ForceLeaderTest.sendDoc(ForceLeaderTest.java:424)
   [junit4]    > 	at org.apache.solr.cloud.ForceLeaderTest.assertSendDocFails(ForceLeaderTest.java:315)
   [junit4]    > 	at org.apache.solr.cloud.ForceLeaderTest.testReplicasInLIRNoLeader(ForceLeaderTest.java:110)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]    > Caused by: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://127.0.0.1:59064/forceleader_test_collection_shard1_replica1]
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:393)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:747)
   [junit4]    > 	... 49 more
   [junit4]    > Caused by: org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://127.0.0.1:59064/forceleader_test_collection_shard1_replica1
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:613)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:413)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:382)
   [junit4]    > 	... 50 more
   [junit4]    > Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 127.0.0.1:59064 [/127.0.0.1] failed: Connection refused
   [junit4]    > 	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151)
   [junit4]    > 	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
   [junit4]    > 	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
   [junit4]    > 	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
   [junit4]    > 	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
   [junit4]    > 	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
   [junit4]    > 	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
   [junit4]    > 	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
   [junit4]    > 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
   [junit4]    > 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:511)
   [junit4]    > 	... 54 more
   [junit4]    > Caused by: java.net.ConnectException: Connection refused
   [junit4]    > 	at java.net.PlainSocketImpl.socketConnect(Native Method)
   [junit4]    > 	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
   [junit4]    > 	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
   [junit4]    > 	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
   [junit4]    > 	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
   [junit4]    > 	at java.net.Socket.connect(Socket.java:589)
   [junit4]    > 	at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:74)
   [junit4]    > 	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:134)

I looked at Jenkins emails from the last week and found the following seeds that also reproduce for me on current master:

ant test  -Dtestcase=ForceLeaderTest -Dtests.method=testReplicasInLIRNoLeader -Dtests.seed=A3BEBFC138E7A44A -Dtests.slow=true -Dtests.asserts=true

ant test  -Dtestcase=ForceLeaderTest -Dtests.method=testReplicasInLIRNoLeader -Dtests.seed=6C380F77C37EF848 -Dtests.multiplier=2 -Dtests.slow=true -Dtests.nightly=true

ant test  -Dtestcase=ForceLeaderTest -Dtests.method=testReplicasInLIRNoLeader -Dtests.seed=32F0374606770814 -Dtests.slow=true -Dtests.asserts=true

ant test  -Dtestcase=ForceLeaderTest -Dtests.method=testReplicasInLIRNoLeader -Dtests.seed=FD60286B08398AC3 -Dtests.slow=true -Dtests.asserts=true

Issue Links

• is related to: SOLR-7569 Create an API to force a leader election between nodes
• relates to: SOLR-9927 revert HttpPartitionTest's avoidance of directUpdatesToLeadersOnly