[SOLR-9390] LeaderFailoverAfterPartitionTest failures: Connection refused, too few active replicas, and no registered leader - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Bug
Status: Open
Priority: Major
Resolution: Unresolved
Affects Version/s: None
Fix Version/s: None
Component/s: None
Labels:
None

Description

In manual testing, I've seen LeaderFailoverAfterPartitionTest fail a couple times, and the seeds reproduce for me:

  [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=LeaderFailoverAfterPartitionTest -Dtests.method=test -Dtests.seed=49939C8CF78131E7 -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=en-AU -Dtests.timezone=America/Port-au-Prince -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   65.8s | LeaderFailoverAfterPartitionTest.test <<<
   [junit4]    > Throwable #1: org.apache.solr.client.solrj.SolrServerException: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://127.0.0.1:60401/c8n_1x3_lf_shard1_replica3]
   [junit4]    >        at __randomizedtesting.SeedInfo.seed([49939C8CF78131E7:C1C7A356597D5C1F]:0)
   [junit4]    >        at org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:774)
   [junit4]    >        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1172)
   [junit4]    >        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1061)
   [junit4]    >        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:997)
   [junit4]    >        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.sendDocsWithRetry(AbstractFullDistribZkTestBase.java:753)
   [junit4]    >        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.sendDocsWithRetry(AbstractFullDistribZkTestBase.java:741)
   [junit4]    >        at org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.testRf3WithLeaderFailover(LeaderFailoverAfterPartitionTest.java:178)
   [junit4]    >        at org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.test(LeaderFailoverAfterPartitionTest.java:57)
   [junit4]    >        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
   [junit4]    >        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
   [junit4]    >        at java.lang.Thread.run(Thread.java:745)
   [junit4]    > Caused by: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://127.0.0.1:60401/c8n_1x3_lf_shard1_replica3]
   [junit4]    >        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:393)
   [junit4]    >        at org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:768)
   [junit4]    >        ... 48 more
   [junit4]    > Caused by: org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://127.0.0.1:60401/c8n_1x3_lf_shard1_replica3
   [junit4]    >        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:615)
   [junit4]    >        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:261)
   [junit4]    >        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
   [junit4]    >        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:413)
   [junit4]    >        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:382)
   [junit4]    >        ... 49 more
   [junit4]    > Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 127.0.0.1:60401 [/127.0.0.1] failed: Connection refused
   [junit4]    >        at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151)
   [junit4]    >        at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
   [junit4]    >        at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
   [junit4]    >        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
   [junit4]    >        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
   [junit4]    >        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
   [junit4]    >        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
   [junit4]    >        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
   [junit4]    >        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
   [junit4]    >        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
   [junit4]    >        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:513)
   [junit4]    >        ... 53 more
   [junit4]    > Caused by: java.net.ConnectException: Connection refused
   [junit4]    >        at java.net.PlainSocketImpl.socketConnect(Native Method)
   [junit4]    >        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
   [junit4]    >        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
   [junit4]    >        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
   [junit4]    >        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
   [junit4]    >        at java.net.Socket.connect(Socket.java:589)
   [junit4]    >        at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:74)
   [junit4]    >        at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:134)
   [junit4]    >        ... 63 more

  [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=LeaderFailoverAfterPartitionTest -Dtests.method=test -Dtests.seed=FEC05A2EE85F1F0B -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=sr-BA -Dtests.timezone=America/Argentina/Salta -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   71.2s | LeaderFailoverAfterPartitionTest.test <<<
   [junit4]    > Throwable #1: org.apache.solr.client.solrj.SolrServerException: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://127.0.0.1:36121/c8n_1x3_lf_shard1_replica3]
   [junit4]    >        at __randomizedtesting.SeedInfo.seed([FEC05A2EE85F1F0B:769465F446A372F3]:0)
   [junit4]    >        at org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:774)
   [junit4]    >        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1172)
   [junit4]    >        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1061)
   [junit4]    >        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:997)
   [junit4]    >        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.sendDocsWithRetry(AbstractFullDistribZkTestBase.java:753)
   [junit4]    >        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.sendDocsWithRetry(AbstractFullDistribZkTestBase.java:741)
   [junit4]    >        at org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.testRf3WithLeaderFailover(LeaderFailoverAfterPartitionTest.java:178)
   [junit4]    >        at org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.test(LeaderFailoverAfterPartitionTest.java:57)
   [junit4]    >        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
   [junit4]    >        at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
   [junit4]    >        at java.lang.Thread.run(Thread.java:745)
   [junit4]    > Caused by: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://127.0.0.1:36121/c8n_1x3_lf_shard1_replica3]
   [junit4]    >        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:393)
   [junit4]    >        at org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:768)
   [junit4]    >        ... 48 more
   [junit4]    > Caused by: org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://127.0.0.1:36121/c8n_1x3_lf_shard1_replica3
   [junit4]    >        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:615)
   [junit4]    >        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:261)
   [junit4]    >        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
   [junit4]    >        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:413)
   [junit4]    >        at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:382)
   [junit4]    >        ... 49 more
   [junit4]    > Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 127.0.0.1:36121 [/127.0.0.1] failed: Connection refused
   [junit4]    >        at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151)
   [junit4]    >        at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
   [junit4]    >        at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
   [junit4]    >        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
   [junit4]    >        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
   [junit4]    >        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
   [junit4]    >        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
   [junit4]    >        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
   [junit4]    >        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
   [junit4]    >        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
   [junit4]    >        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:513)
   [junit4]    >        ... 53 more
   [junit4]    > Caused by: java.net.ConnectException: Connection refused
   [junit4]    >        at java.net.PlainSocketImpl.socketConnect(Native Method)
   [junit4]    >        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
   [junit4]    >        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
   [junit4]    >        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
   [junit4]    >        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
   [junit4]    >        at java.net.Socket.connect(Socket.java:589)
   [junit4]    >        at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:74)
   [junit4]    >        at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:134)
   [junit4]    >        ... 63 more

Attachments

Issue Links

relates to

SOLR-9927 revert HttpPartitionTest's avoidance of directUpdatesToLeadersOnly

Open

Activity

People

Assignee:: Unassigned

Reporter:: Steven Rowe

Votes:: 0 Vote for this issue

Watchers:: 2 Start watching this issue

Dates

Created:: 05/Aug/16 13:30

Updated:: 08/Jun/19 15:37