Major Description
Client makes a HTTP request for a service with "Connection: close" in the HTTP header. When Knox proxies this request as is to the backend service i.e. passing all headers through to the backend service, the backend service honors the "Connection: close" and closes the connection from it's end. In case the backend service does not return "Connection: close" in it's response, Knox HTTP library assumes the connection can be kept alive indefinitely and places it into a pool. Immediate next request for the same resource causes Knox's HTTP library to allocate the pooled connection to it. Since the backend service has closed the connection, this reuse fails and Knox returns a Service connectivity error. The stale connection check by default will not run on every connection request, so requests coming in quickly exacerbate the problem.
In addition I have observed, in some instances the stale connection check will also not help, and so the request fails every alternate time - the first call is where a connection is leased and pooled (though closed already by backend service). The second time the pooled connection is reused and the request fails.
Per https://tools.ietf.org/html/rfc7230#section-6.3 the HTTP library should recognise that a request with "Connection: close" can not possibly leave the connection open so this connection should never be reused/pooled. The latest versions of Knox uses httpcore-4.4.3. From testing and looking at the code, I believe this issue is fixed in httpcore-4.4.6 (`org.apache.http.impl.DefaultConnectionReuseStrategy`).
To fix this issue Knox should move to having httpcore-4.4.6 or higher in it's dependencies. Will attach knox gateway logs that show this problem. The logs are with knox-0.11 but I believe the same problem must exist in knox-0.12 as the same HTTP libraries are used.