[SOLR-4509] Move to non deprecated HttpClient impl classes to remove stale connection check on every request and move connection lifecycle management towards the client.

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 7.0

Environment

5 node SmartOS cluster (all nodes living in same global zone - i.e. same physical machine)

• Assignee: Mark Miller
• Reporter: Ryan Zezeski
• Created: 27/Feb/13 17:51
• Updated: 02/Oct/19 17:24
• Resolved: 08/Apr/16 17:20

Description

By disabling the Apache HTTP Client stale check I've witnessed a 2-4x increase in throughput and reduction of over 100ms. This patch was made in the context of a project I'm leading, called Yokozuna, which relies on distributed search.

Here's the patch on Yokozuna: https://github.com/rzezeski/yokozuna/pull/26

Here's a write-up I did on my findings: http://www.zinascii.com/2013/solr-distributed-search-and-the-stale-check.html

I'm happy to answer any questions or make changes to the patch to make it acceptable.

ReviewBoard: https://reviews.apache.org/r/28393/

Issue Links

• breaks: SOLR-9040 bin/solr SSL support for client->server communcation (as well as default SSL behavior in 'new HttpSolrClient(String)') broken on master
• breaks: SOLR-9262 Connection and read timeouts are being ignored by UpdateShardHandler
• breaks: SOLR-9304 SOLR_SSL_CHECK_PEER_NAME=false (aka -Dsolr.ssl.checkPeerName=false) has no effect
• is duplicated by: SOLR-5604 Remove deprecations caused by httpclient 4.3.x upgrade
• is related to: SOLR-8887 Solr Security features cannot export the internal, deprecated DefaultHttpClient class as part of their user facing API.
• is related to: SOLR-8899 MergeStrategy code creates HttpClient(s) and HttoSolrClient(s) that it does not close
• is related to: SOLR-4328 Simultaneous multiple connections to Solr example often fail with various IOExceptions
• is related to: SOLR-6941 DistributedQueue#containsTaskWithRequestId can fail with NPE.
• is related to: SOLR-6874 There is a race around SocketProxy binding to it's port the way we setup JettySolrRunner and SocketProxy.
• is related to: SOLR-6931 We should do a limited retry when using HttpClient.
• is related to: SOLR-6932 All HttpClient ConnectionManagers and SolrJ clients should always be shutdown in tests and regular code.
• is related to: SOLR-6324 Set finite default timeouts for select and update
• relates to: SOLR-6983 SocketExceptions no longer trigger retries when processing distributed updates
• relates to: SOLR-6793 ReplicationHandler does not destroy all of it's created SnapPullers
• relates to: SOLR-5604 Remove deprecations caused by httpclient 4.3.x upgrade
• relates to: SOLR-10441 remove no longer used HttpShardHandlerFactory.USE_RETRIES