Uploaded image for project: 'jclouds'
  1. jclouds
  2. JCLOUDS-912

GCS uploads with InputStream payloads are not working

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.9.0
    • 2.3.0
    • Ubuntu 14.04.2 LTS 64-bit
      java version "1.7.0_76"
      Java(TM) SE Runtime Environment (build 1.7.0_76-b13)
      Java HotSpot(TM) 64-Bit Server VM (build 24.76-b04, mixed mode)

    Description

      It seems that Payloads based on an InputStream are not currently working with the google-cloud-storage API, either with simpleUpload or multipartUpload. The stream is getting closed before a single byte is read for transmission to the GCS endpoint.

      The payload in the request object is first initialised when GeneratedHttpRequest.Builder.build is called in RestAnnotationProcessor.apply, and then setPayload is later called on the HttpRequest object by MultipartUploadBinder (or UploadBinder in the case of simpleUpload), causing the originally supplied InputStream to be closed.

      The setPayload call in UploadBinder.bindToRequest seems redundant, if the payload has already been set when the request object was built. MultipartUploadBinder.bindToRequest wraps the original payload in a MultipartForm, but the stream in the media part will still end up being closed by the setPayload call. At face value, it doesn't seem like MapBinder.bindToRequest is an appropriate place to call setPayload unless it is legitimate to completely replace the original payload (rather than wrap it) - or at least if it has to mutate the payload property of the request object, then it needs some way of doing it that doesn't result in the originally supplied Payload being 'released'.

      The simplest way of reproducing this is probably just to add a new integration test, similar to ObjectApiLiveTest.testMultipartJpegUpload, but using a FileInputStream based Payload rather than a ByteSource one.

      Relevant portion of stack trace is below.

      Caused by: java.io.IOException: Stream already closed
      	at org.jvnet.mimepull.DataHead$ReadMultiStream.fetch(DataHead.java:248) ~[mimepull-1.9.3.jar:1.9.3]
      	at org.jvnet.mimepull.DataHead$ReadMultiStream.read(DataHead.java:219) ~[mimepull-1.9.3.jar:1.9.3]
      	at java.io.SequenceInputStream.read(SequenceInputStream.java:208) ~[na:1.7.0_76]
      	at java.io.SequenceInputStream.read(SequenceInputStream.java:211) ~[na:1.7.0_76]
      	at java.io.InputStream.read(InputStream.java:101) ~[na:1.7.0_76]
      	at com.google.common.io.ByteStreams.copy(ByteStreams.java:175) ~[guava-16.0.1.jar:na]
      	at org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.writePayloadToConnection(JavaUrlHttpCommandExecutorService.java:297) ~[jclouds-core-1.9.0.jar:1.9.0]
      	at org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:170) ~[jclouds-core-1.9.0.jar:1.9.0]
      	at org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:64) ~[jclouds-core-1.9.0.jar:1.9.0]
      	at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:91) ~[jclouds-core-1.9.0.jar:1.9.0]
      	at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90) ~[jclouds-core-1.9.0.jar:1.9.0]
      	at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73) ~[jclouds-core-1.9.0.jar:1.9.0]
      	at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44) ~[jclouds-core-1.9.0.jar:1.9.0]
      	at org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117) ~[jclouds-core-1.9.0.jar:1.9.0]
      	at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87) ~[guava-16.0.1.jar:na]
      	at com.sun.proxy.$Proxy120.multipartUpload(Unknown Source) ~[na:na]
      

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            gaul Andrew Gaul
            bpiper Ben Piper
            Votes:
            1 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 40m
                40m

                Slack

                  Issue deployment