[SOLR-13036] JettySolrRunner's port retry is broken

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 8.0
• Fix Version/s: 7.7, 8.0
• Component/s: Tests
• Assignee: Gus Heck
• Reporter: Gus Heck
• Created: 03/Dec/18 03:49
• Updated: 08/Jun/19 14:59
• Resolved: 04/Dec/18 03:36

Description

I was running tests and hit the following stack trace which shows the BindException getting wrapped by an IOException, which then evades our port already bound retry logic in JettySolrRunner#retryOnPortBindFailure() which is catching BindException directly. The result is that we never retry as intended. This observation from reading the code is supported by the fact that the logging statement starting with "Port in use..." did not appear in the log.

build.out4.txt- [beaster] [18:06:13.618] ERROR 25.9s J15 | TestCloudConsistency.testOutOfSyncReplicasCannotBecomeLeader <<<
build.out4.txt: [beaster] > Throwable #1: java.io.IOException: Failed to bind to /127.0.0.1:36497
build.out4.txt- [beaster] > at __randomizedtesting.SeedInfo.seed([FD0124C33BA78623:83EA04D3F8C08919]:0)
build.out4.txt- [beaster] > at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:346)
build.out4.txt- [beaster] > at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:308)
build.out4.txt- [beaster] > at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
build.out4.txt- [beaster] > at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:236)
build.out4.txt- [beaster] > at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
build.out4.txt- [beaster] > at org.eclipse.jetty.server.Server.doStart(Server.java:394)
build.out4.txt- [beaster] > at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
build.out4.txt- [beaster] > at org.apache.solr.client.solrj.embedded.JettySolrRunner.retryOnPortBindFailure(JettySolrRunner.java:510)
build.out4.txt- [beaster] > at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:449)
build.out4.txt- [beaster] > at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:417)
build.out4.txt- [beaster] > at org.apache.solr.cloud.TestCloudConsistency.addDocWhenOtherReplicasAreNetworkPartitioned(TestCloudConsistency.java:225)
build.out4.txt- [beaster] > at org.apache.solr.cloud.TestCloudConsistency.testOutOfSyncReplicasCannotBecomeLeader(TestCloudConsistency.java:136)
build.out4.txt- [beaster] > at org.apache.solr.cloud.TestCloudConsistency.testOutOfSyncReplicasCannotBecomeLeader(TestCloudConsistency.java:98)
build.out4.txt- [beaster] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
build.out4.txt- [beaster] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
build.out4.txt- [beaster] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
build.out4.txt- [beaster] > at java.lang.reflect.Method.invoke(Method.java:498)
build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
build.out4.txt- [beaster] > at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
build.out4.txt- [beaster] > at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
build.out4.txt- [beaster] > at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
build.out4.txt- [beaster] > at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
build.out4.txt- [beaster] > at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
build.out4.txt- [beaster] > at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
build.out4.txt- [beaster] > at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
build.out4.txt- [beaster] > at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
build.out4.txt- [beaster] > at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
build.out4.txt- [beaster] > at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
build.out4.txt- [beaster] > at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
build.out4.txt- [beaster] > at java.lang.Thread.run(Thread.java:748)
build.out4.txt- [beaster] > Caused by: java.net.BindException: Address already in use
build.out4.txt- [beaster] > at sun.nio.ch.Net.bind0(Native Method)
build.out4.txt- [beaster] > at sun.nio.ch.Net.bind(Net.java:433)
build.out4.txt- [beaster] > at sun.nio.ch.Net.bind(Net.java:425)
build.out4.txt- [beaster] > at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
build.out4.txt- [beaster] > at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
build.out4.txt- [beaster] > at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:342)
build.out4.txt- [beaster] > ... 51 more

SOLR-7339 first noticed this but seems to have not tracked it down because other bigger problems were extant. 

Issue Links

• is related to: SOLR-12801 Fix the tests, remove BadApples and AwaitsFix annotations, improve env for test development.