Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-9344

BasicAuthIntegrationTest test failures on update

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: trunk
    • Fix Version/s: 6.3
    • Component/s: security, Tests
    • Security Level: Public (Default Security Level. Issues are Public)
    • Labels:
      None

      Description

      I've seen this a number of times while developing SOLR-9200 and SOLR-9324; there's also a public failure here: http://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/17372/

      org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: IOException occured when talking to server at: http://127.0.0.1:45882/solr/testSolrCloudCollection_shard1_replica2
      	at __randomizedtesting.SeedInfo.seed([99BB0D0378978FA8:A463A32F4079D1D8]:0)
      	at org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:760)
      	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1172)
      	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1061)
      	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:997)
      	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
      	at org.apache.solr.security.BasicAuthIntegrationTest.doExtraTests(BasicAuthIntegrationTest.java:193)
      	at org.apache.solr.cloud.TestMiniSolrCloudClusterBase.testCollectionCreateSearchDelete(TestMiniSolrCloudClusterBase.java:196)
      	at org.apache.solr.cloud.TestMiniSolrCloudClusterBase.testBasics(TestMiniSolrCloudClusterBase.java:79)
      	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@9-ea/Native Method)
      	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@9-ea/NativeMethodAccessorImpl.java:62)
      	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@9-ea/DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(java.base@9-ea/Method.java:533)
      	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
      	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
      	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
      	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
      	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
      	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
      	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
      	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
      	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
      	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
      	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
      	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
      	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
      	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
      	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
      	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
      	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
      	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
      	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
      	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
      	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
      	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
      	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
      	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
      	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
      	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
      	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
      	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
      	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
      	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
      	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
      	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
      	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
      	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
      	at java.lang.Thread.run(java.base@9-ea/Thread.java:843)
      Caused by: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://127.0.0.1:45882/solr/testSolrCloudCollection_shard1_replica2
      	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:620)
      	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
      	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
      	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:413)
      	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:366)
      	at org.apache.solr.client.solrj.impl.CloudSolrClient.lambda$directUpdate$0(CloudSolrClient.java:735)
      	at java.util.concurrent.FutureTask.run(java.base@9-ea/FutureTask.java:266)
      	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@9-ea/ThreadPoolExecutor.java:1158)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@9-ea/ThreadPoolExecutor.java:632)
      	... 1 more
      Caused by: org.apache.http.NoHttpResponseException: 127.0.0.1:45882 failed to respond
      	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143)
      	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
      	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
      	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
      	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
      	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
      	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
      	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
      	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
      	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
      	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
      	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
      	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
      	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
      	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:511)
      	... 10 more
      
      1. SOLR-9344.patch
        20 kB
        Alan Woodward
      2. SOLR-9344-httpconfigurer.patch
        16 kB
        Alan Woodward
      3. SOLR-9344-httpconfigurer.patch
        8 kB
        Alan Woodward

        Activity

        Hide
        romseygeek Alan Woodward added a comment -

        I'm pretty sure that this is due to stale connections. Here's a patch that does the following:

        • moves BasicAuthIntegrationTest to use SolrCloudTestCase
        • removes the 'collectionCreateSearchDelete' part of the test, which is duplicated in MiniSolrCloudClusterTest, doesn't seem to do anything useful here, and is the source of any stale connections that are happening
        • remove MiniSolrCloudClusterTestBase, which is now superfluous.
        Show
        romseygeek Alan Woodward added a comment - I'm pretty sure that this is due to stale connections. Here's a patch that does the following: moves BasicAuthIntegrationTest to use SolrCloudTestCase removes the 'collectionCreateSearchDelete' part of the test, which is duplicated in MiniSolrCloudClusterTest, doesn't seem to do anything useful here, and is the source of any stale connections that are happening remove MiniSolrCloudClusterTestBase, which is now superfluous.
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 5b0bf4ab74d18fba9a96a31dbf016b19897adf12 in lucene-solr's branch refs/heads/branch_6x from Alan Woodward
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=5b0bf4a ]

        SOLR-9344: Move BasicAuthIntegrationTest to SolrCloudTestCase

        Show
        jira-bot ASF subversion and git services added a comment - Commit 5b0bf4ab74d18fba9a96a31dbf016b19897adf12 in lucene-solr's branch refs/heads/branch_6x from Alan Woodward [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=5b0bf4a ] SOLR-9344 : Move BasicAuthIntegrationTest to SolrCloudTestCase
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit d0d893e1045eb4d7813e39249ed37b726536eaf3 in lucene-solr's branch refs/heads/master from Alan Woodward
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=d0d893e ]

        SOLR-9344: Move BasicAuthIntegrationTest to SolrCloudTestCase

        Show
        jira-bot ASF subversion and git services added a comment - Commit d0d893e1045eb4d7813e39249ed37b726536eaf3 in lucene-solr's branch refs/heads/master from Alan Woodward [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=d0d893e ] SOLR-9344 : Move BasicAuthIntegrationTest to SolrCloudTestCase
        Hide
        romseygeek Alan Woodward added a comment -

        Will see if this fixes the problem.

        Show
        romseygeek Alan Woodward added a comment - Will see if this fixes the problem.
        Hide
        romseygeek Alan Woodward added a comment -

        Well, we're getting new failures at least: http://jenkins.thetaphi.de/job/Lucene-Solr-6.x-Linux/1679/

        Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        

        Do BasicAuth and SSL not mix Noble Paul? Should we disable SSL on this test class?

        Show
        romseygeek Alan Woodward added a comment - Well, we're getting new failures at least: http://jenkins.thetaphi.de/job/Lucene-Solr-6.x-Linux/1679/ Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target Do BasicAuth and SSL not mix Noble Paul ? Should we disable SSL on this test class?
        Hide
        noble.paul Noble Paul added a comment - - edited

        Alan Woodward I observed a different kind of failure in master. These two are different. SSL and Basic auth should work together perfectly. Actually, we recommend users to setup basicauth with SSL(if possible).

        Show
        noble.paul Noble Paul added a comment - - edited Alan Woodward I observed a different kind of failure in master. These two are different. SSL and Basic auth should work together perfectly. Actually, we recommend users to setup basicauth with SSL(if possible).
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit a0dcf389b2570e341b5c12987d5756b57e7a9b00 in lucene-solr's branch refs/heads/branch_6x from Alan Woodward
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=a0dcf38 ]

        SOLR-9344: Don't reuse port in BasicAuthIntegrationTest

        Show
        jira-bot ASF subversion and git services added a comment - Commit a0dcf389b2570e341b5c12987d5756b57e7a9b00 in lucene-solr's branch refs/heads/branch_6x from Alan Woodward [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=a0dcf38 ] SOLR-9344 : Don't reuse port in BasicAuthIntegrationTest
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 003a346d5005afad20a1766bafbffa18621d9d62 in lucene-solr's branch refs/heads/master from Alan Woodward
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=003a346 ]

        SOLR-9344: Don't reuse port in BasicAuthIntegrationTest

        Show
        jira-bot ASF subversion and git services added a comment - Commit 003a346d5005afad20a1766bafbffa18621d9d62 in lucene-solr's branch refs/heads/master from Alan Woodward [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=003a346 ] SOLR-9344 : Don't reuse port in BasicAuthIntegrationTest
        Hide
        romseygeek Alan Woodward added a comment -

        OK, this looks like a genuine bug. The above exception is caused when trying to use an HttpClient configured by the PKIAuthenticationPlugin, in combination with SSL. Tests using SSL set a static HttpClientConfigurer on HttpClientUtil, but this is then replaced by the PKIAuthenticationPlugin, so any clients created after this replacement don't get SSL support. Maybe HttpClientUtil needs to maintain a list of configurers and run through them all, rather than just having a single one?

        This only happens on 6.x, as the mechanism has changed in master.

        Show
        romseygeek Alan Woodward added a comment - OK, this looks like a genuine bug. The above exception is caused when trying to use an HttpClient configured by the PKIAuthenticationPlugin, in combination with SSL. Tests using SSL set a static HttpClientConfigurer on HttpClientUtil, but this is then replaced by the PKIAuthenticationPlugin, so any clients created after this replacement don't get SSL support. Maybe HttpClientUtil needs to maintain a list of configurers and run through them all, rather than just having a single one? This only happens on 6.x, as the mechanism has changed in master.
        Hide
        noble.paul Noble Paul added a comment -

        Good catch. What's stopping us from moving the coffee from master to 6.x

        Show
        noble.paul Noble Paul added a comment - Good catch. What's stopping us from moving the coffee from master to 6.x
        Hide
        romseygeek Alan Woodward added a comment -

        Looking at SOLR-9028 and SOLR-4509, it looks like it we can't backport the mechanism changes without breaking the auth plugin API. Instead, here's a patch for 6x only that changes HttpClientUtil to hold a list of HttpClientConfigurer objects, which seems to make tests pass. Hoss Man Mark Miller you did most of the work changing stuff for the master branch, does this change make sense?

        Show
        romseygeek Alan Woodward added a comment - Looking at SOLR-9028 and SOLR-4509 , it looks like it we can't backport the mechanism changes without breaking the auth plugin API. Instead, here's a patch for 6x only that changes HttpClientUtil to hold a list of HttpClientConfigurer objects, which seems to make tests pass. Hoss Man Mark Miller you did most of the work changing stuff for the master branch, does this change make sense?
        Hide
        romseygeek Alan Woodward added a comment -

        Hm, actually scratch this patch, it's causing a whole bag of test failures, looking into why now...

        Show
        romseygeek Alan Woodward added a comment - Hm, actually scratch this patch, it's causing a whole bag of test failures, looking into why now...
        Hide
        romseygeek Alan Woodward added a comment -

        Fixed the test failures - I added a 'resetConfigurers()' method to HttpClientUtil which removes all added configurers and goes back to using the default, and also ensured that a default configurer was always the first element in the chain.

        Show
        romseygeek Alan Woodward added a comment - Fixed the test failures - I added a 'resetConfigurers()' method to HttpClientUtil which removes all added configurers and goes back to using the default, and also ensured that a default configurer was always the first element in the chain.
        Hide
        noble.paul Noble Paul added a comment -

        Alan Woodward this patch is for branch_6x right ?

        Show
        noble.paul Noble Paul added a comment - Alan Woodward this patch is for branch_6x right ?
        Hide
        romseygeek Alan Woodward added a comment -

        Right, this is for 6.x only, master isn't affected by this bug (and the test failures are all appearing on the 6x branch tests).

        Show
        romseygeek Alan Woodward added a comment - Right, this is for 6.x only, master isn't affected by this bug (and the test failures are all appearing on the 6x branch tests).
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 66a51b76b399ac0b018a4f600de3fc349ae7a2ec in lucene-solr's branch refs/heads/branch_6x from Alan Woodward
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=66a51b7 ]

        SOLR-9344: Allow multiple HttpClientConfigurers to be set on HttpClientUtil

        Show
        jira-bot ASF subversion and git services added a comment - Commit 66a51b76b399ac0b018a4f600de3fc349ae7a2ec in lucene-solr's branch refs/heads/branch_6x from Alan Woodward [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=66a51b7 ] SOLR-9344 : Allow multiple HttpClientConfigurers to be set on HttpClientUtil
        Hide
        romseygeek Alan Woodward added a comment -

        Looks like this has finally been fixed!

        Show
        romseygeek Alan Woodward added a comment - Looks like this has finally been fixed!
        Hide
        shalinmangar Shalin Shekhar Mangar added a comment -

        Closing after 6.3.0 release.

        Show
        shalinmangar Shalin Shekhar Mangar added a comment - Closing after 6.3.0 release.

          People

          • Assignee:
            romseygeek Alan Woodward
            Reporter:
            gchanan Gregory Chanan
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development