[SOLR-14456] Compressed requests fail in SolrCloud when the request is routed internally by the serving solr node

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 7.7.2
• Fix Version/s: 8.6, 9.0
• Component/s: SolrCloud

Environment

Solr version: 7.7.2

Solr cloud enabled

Cluster topology: 6 nodes, 1 single collection, 10 shards and 3 replicas. 1 HTTP LB using Round Robin over all nodes

All cluster nodes have gzip enabled for all paths, all HTTP verbs and all MIME types.

Solr client: HttpSolrClient targeting the HTTP LB

 

• Assignee: Houston Putman
• Reporter: Samuel García Martínez
• Created: 02/May/20 09:11
• Updated: 15/Jul/20 15:09
• Resolved: 13/May/20 19:32

Description

Solr cluster setup

• Solr version: 7.7.2
• Solr cloud enabled
• Cluster topology: 6 nodes, 1 single collection, 10 shards and 3 replicas. 1 HTTP LB using Round Robin over all nodes
• All cluster nodes have gzip enabled for all paths, all HTTP verbs and all MIME types.
• Solr client: HttpSolrClient targeting the HTTP LB

Problem description

When the Solr node that receives the request has to forward it
to a Solr Node that can actually perform the query, the response headers are added incorrectly to the response, causing any HTTP client to fail, whether it's a SolrClient or a basic HTTP client implementation with any other SDK.

To simplify the case, let's try to start from the following repro scenario:

• Start one node with cloud mode and port 8983
• Create one single collection (1 shard, 1 replica)
• Start another node with port 8984 and the previusly started zk (-z localhost:9983)
• Start a java application and query the cluster using the node on port 8984 (the one that doesn't host the collection)

So, then something like this happens:

• The application queries node:8984 with compression enabled ("Accept-Encoding: gzip")
  and wt=javabin
• Node:8984 can't perform the query and creates a http request behind the scenes to node:8983
• Node:8983 returns a gzipped response with "Content-Encoding: gzip" and "Content-Type:
  application/octet-stream"
  Node:8984 adds the "Content-Encoding: gzip" header as character stream to the response
  (it should be forwarded as "Content-Encoding" header, not character encoding)
• HttpSolrClient receives a "Content-Type: application/octet-stream;charset=gzip", causing
  an exception.
• HttpSolrClient tries to quietly close the connection, but since the stream is broken,
  the Utils.consumeFully fails to actually consume the entity (it throws another exception in
  GzipDecompressingEntity#getContent() with "not in GZIP format")

The exception thrown by HttpSolrClient is:

java.nio.charset.UnsupportedCharsetException: gzip
 at java.nio.charset.Charset.forName(Charset.java:531)
 at org.apache.http.entity.ContentType.create(ContentType.java:271)
 at org.apache.http.entity.ContentType.create(ContentType.java:261)
 at org.apache.http.entity.ContentType.parse(ContentType.java:319)
 at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:591)
 at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
 at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
 at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
 at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:1015)
 at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:1031)
 at org.apache.solr.client.solrj.SolrClient$$FastClassBySpringCGLIB$$7fcf36a0.invoke(<generated>)
 at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)

 

Issue Links

• relates to: SOLR-14457 SolrClient leaks connections on compressed responses if the response is malformed
• links to: GitHub Pull Request #1480