[HTTPCLIENT-1992] Impossible to access trailer-headers available in chunked transfer-encoding

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 5.0 Beta4, 5.0 Beta5
• Fix Version/s: 5.0 Beta5
• Component/s: HttpClient (classic)
• Assignee: Unassigned
• Reporter: Serkan Turgut
• Created: 11/Jun/19 18:19
• Updated: 15/Mar/23 17:17
• Resolved: 14/Jun/19 09:43

Description

Message trailers in chunked messaging are being parsed and kept in ChunkedInputStream. The parsed trailer headers are exposed with getFooters() method. However, there is no caller of this method in the entire call stack during a chunked messaging (both in version 4 and 5).

HttpClient 5 brings the support for message trailers by creating an API in HttpEntity interface named as getTrailers().). Even though there are multiple implementations of these HttpEntity, none of them reach to ChunkedInputStream.getFooters() to get the trailers and expose them to caller of HttpEntity.getTrailers().

• AbstractHttpEntity

• • Following implementations return hardcoded 'null' for getTrailers() because the default implementation in AbstractHttpEntity returns null
  • BasicHttpEntity
  • BufferedHttpEntity
  • ByteArrayEntity
  • ByteBufferEntity
  • EntityTemplate
  • FileEntity
  • InputStreamEntity
  • SerializableEntity
  • StringEntity
  • ResponseEntityProxy
  • HttpEntityWithTrailers
    • Only returns the trailer list given its constructor instead of getting them from ChunkedInputStream.getFooters()
  • HttpEntityWrapper
    • Call wrappterEntity.getTrailers(), if the underlying entity is one of them in this list, this class also returns null

Currently the call chain is following (omitted irrelevant calls);
#1 MainClientExec.execute() -> creates ResponseEntityProxy regardless the response is chunked or not.
#2 ResponseEntityProxy.enhance() -> wraps the underlying stream with EofSensorInputStream which hides ChunkedInputStream.getFooters() method from outside. Since EofSensorInputStream is unaware of chunked messaging and trailers, there is no way to reach ChuckedInputStream.getFooters() from outside after this point.
#3 EofSensorInputStream -> This is to watch the stream and free it if it's EOF.
#4 ChunkedInputStream.read()
#5 ChunkedInputStream.nextChunk()
#6 ChunkedInputStream.parseTrailerHeaders() -> Parses the trailer headers and puts into ChunkedInputSteam.footers field, exposed by getFooters() method but no one is calling this method.

Code snip to observe the behavior;

   CloseableHttpClient httpClient = HttpClients.custom().addResponseInterceptorLast(new HttpResponseInterceptor() {
            @Override
            public void process(HttpResponse response, EntityDetails entity, HttpContext context) throws HttpException, IOException {
                if (response instanceof CloseableHttpResponse) {
                    CloseableHttpResponse closeableHttpResponse = (CloseableHttpResponse) response;
                    HttpEntity httpEntity = closeableHttpResponse.getEntity();
                    Supplier<List<? extends Header>> trailers = httpEntity.getTrailers();
                    // --> 'trailers' is always NULL
                }
            }
        }).build();

// Make Http requests to any server which can responds in 'Transfer-Encoding: chunked' and send trailers, observe the trailer supplier above is always null.

Proposed solution (the patch is also ready, I will submit a pull request if agree on the problem and proposal)

There needs to be a way to propagate what has been parsed in ChunkedInputStream.parseTrailerHeaders() by calling ChunkedInputStream.getFooters() and pass those headers to callers of HttpEntity.getTrailers(). To overcome this issue, the proposal is the following;

• Implement a version of ResponseEntityProxy named as ResponseChunkedEntityProxy for chunked messaging so it recognizes the underlying stream is ChunkedInputStream and It can return a Supplier which delegates the call to ChunkedInputStream.getFooters() when ResponseChunkedEntityProxy.getTrailers() called and return the supplier for trailers to be used by users of HttpClient at the end of the communication.
• Change MainClientExec.execute in a way to use ResponseChunkedEntityProxy when the response header 'Transfer-Encoding: chunked' presents, so the HttpEntity.getTrailers() API can return trailers parsed by ChunkedInputStream.

Issue Links

• is related to: HTTPCLIENT-2263 Cannot Access Trailer-Headers in Chunked transfer-encoding
• links to: GitHub Pull Request #153