[JCLOUDS-532] Potential for read on closed stream when handling HTTP 5xx errors

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.8.0
• Fix Version/s: 1.9.0, 2.0.0
• Component/s: jclouds-core
• Assignee: Ignasi Barrera
• Reporter: Jasdeep Hundal
• Created: 11/Apr/14 00:07
• Updated: 12/Sep/16 21:15
• Resolved: 10/Mar/15 21:09

Description

I've seen this traceback when testing JClouds against a broken Cinder endpoint (though I suspect something similar is possible in other APIs):

Caused by: org.jclouds.http.HttpResponseException: java.io.IOException: Attempted read on closed stream. connecting to DELETE http://192.168.56.20:8776/v1/c33361f3a390470c918276fdd50795ba /volumes/d77bf972-3917-43e8-85dd-894611fcdc4d HTTP/1.1
    	at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:162)
    	at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:93)
    	at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:76)
    	at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:47)
    	at org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
    	at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
    	at com.sun.proxy.$Proxy98.delete(Unknown Source)
    	... 49 more
    Caused by: java.lang.RuntimeException: java.io.IOException: Attempted read on closed stream.
    	at com.google.common.base.Throwables.propagate(Throwables.java:160)
    	at org.jclouds.http.HttpUtils.toByteArrayOrNull(HttpUtils.java:131)
    	at org.jclouds.http.HttpUtils.closeClientButKeepContentStream(HttpUtils.java:163)
    	at org.jclouds.openstack.cinder.v1.handlers.CinderErrorHandler.handleError(CinderErrorHandler.java:42)
    	at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:67)
    	at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:180)
    	at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:150)
    	... 56 more
    Caused by: java.io.IOException: Attempted read on closed stream.
    	at org.apache.http.conn.EofSensorInputStream.isReadAllowed(EofSensorInputStream.java:109)
    	at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:135)
    	at java.io.FilterInputStream.read(FilterInputStream.java:133)
    	at java.io.FilterInputStream.read(FilterInputStream.java:107)
    	at com.google.common.io.ByteStreams.copy(ByteStreams.java:175)
    	at com.google.common.io.ByteStreams.toByteArray(ByteStreams.java:220)
    	at org.jclouds.http.HttpUtils.toByteArrayOrNull(HttpUtils.java:129)
    	... 61 more

This seems possible if the payload in an HttpResponse is an InputStreamPayload and the HTTP status code received is a 5xx.
The BackoffLimitedRetryHandler can hit this line to release the payload from a request, which will close the stream: https://github.com/jclouds/jclouds/blob/master/core/src/main/java/org/jclouds/http/handlers/BackoffLimitedRetryHandler.java#L99
Then the CinderErrorHandler attempts to read the stream here after the HttpResponse is passed off to it:
https://github.com/jclouds/jclouds/blob/master/apis/openstack-cinder/src/main/java/org/jclouds/openstack/cinder/v1/handlers/CinderErrorHandler.java#L42

I would like to be able to write a test to reproduce this, but it doesn't seem that I can set the payload correctly in the VolumeApiExpectTest to trigger this. (I get a null payload instead of an InputStreamPayload when trying to build an HttpResponse with the latter.)

Issue Links

• breaks: JCLOUDS-1176 Swift delete operation is not idempotent