Solr
  1. Solr
  2. SOLR-4451

Upgrade to httpclient 4.2.x and take advantage of SystemDefaultHttpClient

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.2, 6.0
    • Component/s: None
    • Labels:
      None

      Description

      HttpComponent is up to version 4.2, and included in 4.2 is a new subclass of DefaultHttpClient named SystemDefaultHttpClient, which automatically configures itself using the "standard" java system properties...

      http://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/impl/client/SystemDefaultHttpClient.html

      ...i think we should upgrade and start using this new class in place of DefaultHttpClient, so that SolrJ clients (and implicitly SolrCloud) can automaticly leverage system properties users may expect to work.

        Issue Links

          Activity

          Hide
          Hoss Man added a comment -

          Patch...

          • upgrades httpcomponents in ivy
          • upgrades httpcomponents in maven pom (i think)
          • switches from DefaultHttpClient to SystemDefaultHttpClient in solrj
          • fixes some existing helper code that treats ThreadSafeClientConnManager as special to also work with PoolingClientConnectionManager (all ClientConnectionManager impls are now thread safe, and ThreadSafeClientConnManager has been deprecated with PoolingClientConnectionManager as the recommended replacement)

          ...test and precommit all pass

          Show
          Hoss Man added a comment - Patch... upgrades httpcomponents in ivy upgrades httpcomponents in maven pom (i think) switches from DefaultHttpClient to SystemDefaultHttpClient in solrj fixes some existing helper code that treats ThreadSafeClientConnManager as special to also work with PoolingClientConnectionManager (all ClientConnectionManager impls are now thread safe, and ThreadSafeClientConnManager has been deprecated with PoolingClientConnectionManager as the recommended replacement) ...test and precommit all pass
          Hide
          Hoss Man added a comment -

          This should help simplify a lot of things relating to "customizing" how solr utilizes httpclient – in particular dealing with basic auth and SSL (especially testing SSL with non standard key/trust stores)

          Show
          Hoss Man added a comment - This should help simplify a lot of things relating to "customizing" how solr utilizes httpclient – in particular dealing with basic auth and SSL (especially testing SSL with non standard key/trust stores)
          Hide
          Hoss Man added a comment -

          Committed revision 1445945.
          Committed revision 1445955.

          Show
          Hoss Man added a comment - Committed revision 1445945. Committed revision 1445955.
          Hide
          Commit Tag Bot added a comment -

          [branch_4x commit] Chris M. Hostetter
          http://svn.apache.org/viewvc?view=revision&revision=1445955

          SOLR-4451: SolrJ, and SolrCloud internals, now use SystemDefaultHttpClient under the covers – allowing many HTTP connection related properties to be controlled via 'standard' java system properties. (merge r1445945)

          Show
          Commit Tag Bot added a comment - [branch_4x commit] Chris M. Hostetter http://svn.apache.org/viewvc?view=revision&revision=1445955 SOLR-4451 : SolrJ, and SolrCloud internals, now use SystemDefaultHttpClient under the covers – allowing many HTTP connection related properties to be controlled via 'standard' java system properties. (merge r1445945)
          Hide
          Commit Tag Bot added a comment -

          [trunk commit] Chris M. Hostetter
          http://svn.apache.org/viewvc?view=revision&revision=1445945

          SOLR-4451: SolrJ, and SolrCloud internals, now use SystemDefaultHttpClient under the covers – allowing many HTTP connection related properties to be controlled via 'standard' java system properties. (hossman)

          Show
          Commit Tag Bot added a comment - [trunk commit] Chris M. Hostetter http://svn.apache.org/viewvc?view=revision&revision=1445945 SOLR-4451 : SolrJ, and SolrCloud internals, now use SystemDefaultHttpClient under the covers – allowing many HTTP connection related properties to be controlled via 'standard' java system properties. (hossman)
          Hide
          Hoss Man added a comment -

          FYI: my upgrade broke the maven build because i had the transative deps wrong, fixed in SOLR-4462

          Show
          Hoss Man added a comment - FYI: my upgrade broke the maven build because i had the transative deps wrong, fixed in SOLR-4462
          Hide
          Mark Miller added a comment -

          Some tests run from eclipse now seem to fail after this. Is there a sys prop I need to pass now or something? I seem to get:

          410 T11 oasc.SolrException.log SEVERE null:java.lang.NoSuchMethodError: org.apache.http.impl.conn.SchemeRegistryFactory.createSystemDefault()Lorg/apache/http/conn/scheme/SchemeRegistry;
          		at org.apache.http.impl.client.SystemDefaultHttpClient.createClientConnectionManager(SystemDefaultHttpClient.java:118)
          		at org.apache.http.impl.client.AbstractHttpClient.getConnectionManager(AbstractHttpClient.java:445)
          		at org.apache.solr.client.solrj.impl.HttpClientUtil.setMaxConnections(HttpClientUtil.java:179)
          		at org.apache.solr.client.solrj.impl.HttpClientConfigurer.configure(HttpClientConfigurer.java:33)
          		at org.apache.solr.client.solrj.impl.HttpClientUtil.configureClient(HttpClientUtil.java:115)
          		at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:105)
          		at org.apache.solr.handler.component.HttpShardHandlerFactory.init(HttpShardHandlerFactory.java:134)
          		at org.apache.solr.core.CoreContainer.initShardHandler(CoreContainer.java:704)
          Show
          Mark Miller added a comment - Some tests run from eclipse now seem to fail after this. Is there a sys prop I need to pass now or something? I seem to get: 410 T11 oasc.SolrException.log SEVERE null:java.lang.NoSuchMethodError: org.apache.http.impl.conn.SchemeRegistryFactory.createSystemDefault()Lorg/apache/http/conn/scheme/SchemeRegistry; at org.apache.http.impl.client.SystemDefaultHttpClient.createClientConnectionManager(SystemDefaultHttpClient.java:118) at org.apache.http.impl.client.AbstractHttpClient.getConnectionManager(AbstractHttpClient.java:445) at org.apache.solr.client.solrj.impl.HttpClientUtil.setMaxConnections(HttpClientUtil.java:179) at org.apache.solr.client.solrj.impl.HttpClientConfigurer.configure(HttpClientConfigurer.java:33) at org.apache.solr.client.solrj.impl.HttpClientUtil.configureClient(HttpClientUtil.java:115) at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:105) at org.apache.solr.handler.component.HttpShardHandlerFactory.init(HttpShardHandlerFactory.java:134) at org.apache.solr.core.CoreContainer.initShardHandler(CoreContainer.java:704)
          Hide
          Hoss Man added a comment -

          mark: nothing should be intrinsically dependent on a system property, it just uses them if they are set to provide defaults.

          410 T11 oasc.SolrException.log SEVERE null:java.lang.NoSuchMethodError: org.apache.http.impl.conn.SchemeRegistryFactory.createSystemDefault()Lorg/apache/http/conn/scheme/SchemeRegistry;

          Do you see this before or after sarowe's changes in SOLR-4462 (or both?)

          Show
          Hoss Man added a comment - mark: nothing should be intrinsically dependent on a system property, it just uses them if they are set to provide defaults. 410 T11 oasc.SolrException.log SEVERE null:java.lang.NoSuchMethodError: org.apache.http.impl.conn.SchemeRegistryFactory.createSystemDefault()Lorg/apache/http/conn/scheme/SchemeRegistry; Do you see this before or after sarowe's changes in SOLR-4462 (or both?)
          Hide
          Hoss Man added a comment -

          mark: Is there a classpath conflict between the httpclient version used by solr and some other httpclient version used in eclipse? because "
          org.apache.http.impl.conn.SchemeRegistryFactory.createSystemDefault()Lorg/apache/http/conn/scheme/SchemeRegistry" seems to exist in ./solrj/lib/httpclient-4.2.3.jar (which didn't change as part of SOLR-4462)

          Show
          Hoss Man added a comment - mark: Is there a classpath conflict between the httpclient version used by solr and some other httpclient version used in eclipse? because " org.apache.http.impl.conn.SchemeRegistryFactory.createSystemDefault()Lorg/apache/http/conn/scheme/SchemeRegistry" seems to exist in ./solrj/lib/httpclient-4.2.3.jar (which didn't change as part of SOLR-4462 )
          Hide
          Robert Muir added a comment -

          i did the following:

          svn up
          ant eclipse
          refresh project

          and tests seemed to work fine here

          Show
          Robert Muir added a comment - i did the following: svn up ant eclipse refresh project and tests seemed to work fine here
          Hide
          Mark Miller added a comment -

          No such luck for me - I'll be trying to figure out why later.

          Show
          Mark Miller added a comment - No such luck for me - I'll be trying to figure out why later.
          Hide
          Ursula Kellmann added a comment -

          Hello,

          After upgrading from 4.1 to 4.2 our api to create a HttpSolrServer no longer worked. Our unit tests using the embedded solr server did not catch this issue. We did not detect the issue until runtime with deployment to jboss 7.
          The errors were
          java.io.FileNotFoundException: \local\java\jre\lib\security\cacerts (The system cannot find the path specified)
          Failure initializing default system SSL context

          In our environment we do not require security to access the solr server. To get around this error, we created the http client and set the connection manager on it and passing it to the HttpSolrServer constructor.
          i.e.
          PoolingClientConnectionManager connManager = new PoolingClientConnectionManager();
          DefaultHttpClient client = new DefaultHttpClient(connManager);

          Can you please advise if this is the best way to get around this? Also, is there a way that a unit test could have found this issue?

          Thank you in advance for for any help you can provide, and we think solr is awesome!

          Regards,
          Ursula Kellmann

          Show
          Ursula Kellmann added a comment - Hello, After upgrading from 4.1 to 4.2 our api to create a HttpSolrServer no longer worked. Our unit tests using the embedded solr server did not catch this issue. We did not detect the issue until runtime with deployment to jboss 7. The errors were java.io.FileNotFoundException: \local\java\jre\lib\security\cacerts (The system cannot find the path specified) Failure initializing default system SSL context In our environment we do not require security to access the solr server. To get around this error, we created the http client and set the connection manager on it and passing it to the HttpSolrServer constructor. i.e. PoolingClientConnectionManager connManager = new PoolingClientConnectionManager(); DefaultHttpClient client = new DefaultHttpClient(connManager); Can you please advise if this is the best way to get around this? Also, is there a way that a unit test could have found this issue? Thank you in advance for for any help you can provide, and we think solr is awesome! Regards, Ursula Kellmann
          Hide
          Ken Krugler added a comment -

          One of my developers also ran into what seems like the same issue, when trying to use the embedded Solr server. Worked fine in unit tests, but gets a NoSuchMethodError when running on the Hadoop cluster. Whereas 4.1 works fine. I'm assuming a classpath issue with some Hadoop jars, so more research is needed, but wanted to add to the discussion above.

          Show
          Ken Krugler added a comment - One of my developers also ran into what seems like the same issue, when trying to use the embedded Solr server. Worked fine in unit tests, but gets a NoSuchMethodError when running on the Hadoop cluster. Whereas 4.1 works fine. I'm assuming a classpath issue with some Hadoop jars, so more research is needed, but wanted to add to the discussion above.
          Hide
          Ken Krugler added a comment -

          One related question - if I'm using embedded Solr, is there some way to load a core without initShardHandler() being called? From looking at the code, that's not possible, but seems like something that should be configurable (similar to how ZooKeeper support is conditional on some system properties).

          Show
          Ken Krugler added a comment - One related question - if I'm using embedded Solr, is there some way to load a core without initShardHandler() being called? From looking at the code, that's not possible, but seems like something that should be configurable (similar to how ZooKeeper support is conditional on some system properties).
          Hide
          Anand added a comment -

          Upgraded to Solr 4.2.1 today. Also, updated http components to the ones provided in the package (httpclient-4.2.3 and httpcore-4.2.2) and I was still seeing the error message "java.lang.NoSuchMethodError: org.apache.http.impl.conn.SchemeRegistryFactory.createSystemDefault()Lorg/apache/http/conn/scheme/SchemeRegistry;" Turns out the httpclient jar was conflicting with httpclient jar used by selenium tests. Once they were aligned, things started looking a lot better.

          Show
          Anand added a comment - Upgraded to Solr 4.2.1 today. Also, updated http components to the ones provided in the package (httpclient-4.2.3 and httpcore-4.2.2) and I was still seeing the error message "java.lang.NoSuchMethodError: org.apache.http.impl.conn.SchemeRegistryFactory.createSystemDefault()Lorg/apache/http/conn/scheme/SchemeRegistry;" Turns out the httpclient jar was conflicting with httpclient jar used by selenium tests. Once they were aligned, things started looking a lot better.
          Hide
          Uwe Schindler added a comment -

          Closed after release.

          Show
          Uwe Schindler added a comment - Closed after release.
          Hide
          Grant Ingersoll added a comment -

          Ken Krugler did you ever get this straightened out for Hadoop? I'm hitting it now, w/ the issue being that Hadoop has an older version of HttpClient in it's lib.

          Show
          Grant Ingersoll added a comment - Ken Krugler did you ever get this straightened out for Hadoop? I'm hitting it now, w/ the issue being that Hadoop has an older version of HttpClient in it's lib.
          Hide
          Grant Ingersoll added a comment -

          Sure would be nice to do something like: https://maven.apache.org/plugins/maven-shade-plugin/examples/class-relocation.html on this. If only there were an Ant plugin for it.

          Show
          Grant Ingersoll added a comment - Sure would be nice to do something like: https://maven.apache.org/plugins/maven-shade-plugin/examples/class-relocation.html on this. If only there were an Ant plugin for it.
          Show
          Grant Ingersoll added a comment - A little googling leads to http://one-jar.sourceforge.net/index.php?page=introduction&file=intro
          Hide
          Ken Krugler added a comment -

          Grant Ingersoll yes we got it to work (this was in EMR) by creating a custom bootstrap action script that "fixed up" the set of jars in the Hadoop lib dir to be in sync with what Solr was/is using.

          Show
          Ken Krugler added a comment - Grant Ingersoll yes we got it to work (this was in EMR) by creating a custom bootstrap action script that "fixed up" the set of jars in the Hadoop lib dir to be in sync with what Solr was/is using.

            People

            • Assignee:
              Hoss Man
              Reporter:
              Hoss Man
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development