Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-13036

JettySolrRunner's port retry is broken

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 8.0
    • Fix Version/s: 7.7, 8.0
    • Component/s: Tests
    • Labels:
      None

      Description

      I was running tests and hit the following stack trace which shows the BindException getting wrapped by an IOException, which then evades our port already bound retry logic in JettySolrRunner#retryOnPortBindFailure() which is catching BindException directly. The result is that we never retry as intended. This observation from reading the code is supported by the fact that the logging statement starting with "Port in use..." did not appear in the log.

      build.out4.txt- [beaster] [18:06:13.618] ERROR 25.9s J15 | TestCloudConsistency.testOutOfSyncReplicasCannotBecomeLeader <<<
      build.out4.txt: [beaster] > Throwable #1: java.io.IOException: Failed to bind to /127.0.0.1:36497
      build.out4.txt- [beaster] > at __randomizedtesting.SeedInfo.seed([FD0124C33BA78623:83EA04D3F8C08919]:0)
      build.out4.txt- [beaster] > at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:346)
      build.out4.txt- [beaster] > at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:308)
      build.out4.txt- [beaster] > at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
      build.out4.txt- [beaster] > at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:236)
      build.out4.txt- [beaster] > at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
      build.out4.txt- [beaster] > at org.eclipse.jetty.server.Server.doStart(Server.java:394)
      build.out4.txt- [beaster] > at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
      build.out4.txt- [beaster] > at org.apache.solr.client.solrj.embedded.JettySolrRunner.retryOnPortBindFailure(JettySolrRunner.java:510)
      build.out4.txt- [beaster] > at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:449)
      build.out4.txt- [beaster] > at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:417)
      build.out4.txt- [beaster] > at org.apache.solr.cloud.TestCloudConsistency.addDocWhenOtherReplicasAreNetworkPartitioned(TestCloudConsistency.java:225)
      build.out4.txt- [beaster] > at org.apache.solr.cloud.TestCloudConsistency.testOutOfSyncReplicasCannotBecomeLeader(TestCloudConsistency.java:136)
      build.out4.txt- [beaster] > at org.apache.solr.cloud.TestCloudConsistency.testOutOfSyncReplicasCannotBecomeLeader(TestCloudConsistency.java:98)
      build.out4.txt- [beaster] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      build.out4.txt- [beaster] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      build.out4.txt- [beaster] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      build.out4.txt- [beaster] > at java.lang.reflect.Method.invoke(Method.java:498)
      build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
      build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
      build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
      build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
      build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
      build.out4.txt- [beaster] > at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
      build.out4.txt- [beaster] > at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
      build.out4.txt- [beaster] > at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
      build.out4.txt- [beaster] > at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
      build.out4.txt- [beaster] > at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
      build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
      build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
      build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
      build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
      build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
      build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
      build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
      build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
      build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
      build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
      build.out4.txt- [beaster] > at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
      build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
      build.out4.txt- [beaster] > at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
      build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
      build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
      build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
      build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
      build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
      build.out4.txt- [beaster] > at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
      build.out4.txt- [beaster] > at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
      build.out4.txt- [beaster] > at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
      build.out4.txt- [beaster] > at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
      build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
      build.out4.txt- [beaster] > at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
      build.out4.txt- [beaster] > at java.lang.Thread.run(Thread.java:748)
      build.out4.txt- [beaster] > Caused by: java.net.BindException: Address already in use
      build.out4.txt- [beaster] > at sun.nio.ch.Net.bind0(Native Method)
      build.out4.txt- [beaster] > at sun.nio.ch.Net.bind(Net.java:433)
      build.out4.txt- [beaster] > at sun.nio.ch.Net.bind(Net.java:425)
      build.out4.txt- [beaster] > at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
      build.out4.txt- [beaster] > at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
      build.out4.txt- [beaster] > at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:342)
      build.out4.txt- [beaster] > ... 51 more
      
      

      SOLR-7339 first noticed this but seems to have not tracked it down because other bigger problems were extant. 

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                gus_heck Gus Heck
                Reporter:
                gus_heck Gus Heck
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: