Uploaded image for project: 'jclouds'
  1. jclouds
  2. JCLOUDS-1472

Multipart upload for AzureBlob fails when using InputStream for files greater than 32MB

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.1.0, 2.1.1
    • Fix Version/s: 2.2.0, 2.1.2
    • Component/s: jclouds-blobstore
    • Labels:
      None
    • Environment:
      Java 8 (191)
      Windows 10

      Description

      There are two things to mention.

      1. When uploading a big file we use multipart for "supposed" efficiency but we found it only works when payload is passed as s `File`. Any kind of InputStream fails with the message

      org.jclouds.http.HttpResponseException: Stream closed connecting to PUT https://__/__/myfile_100MB.bin-20181208-074010dic-4361f219-88c1-48cf-8e06-a9cd3035a05e?comp=block&blockid=AAAAAw%3D%3D HTTP/1.1
      	at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:120)
      	at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:91)
      	at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74)
      	at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:45)
      	at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
      	at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
      	at com.sun.proxy.$Proxy44.putBlock(Unknown Source)
      	at org.jclouds.azureblob.blobstore.AzureBlobStore.uploadMultipartPart(AzureBlobStore.java:446)
      	at org.jclouds.blobstore.internal.BaseBlobStore$BlobUploader.call(BaseBlobStore.java:396)
      	at org.jclouds.blobstore.internal.BaseBlobStore$BlobUploader.call(BaseBlobStore.java:383)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      	at java.lang.Thread.getStackTrace(Thread.java:1559)
      	at org.jclouds.blobstore.internal.BaseBlobStore.putMultipartBlob(BaseBlobStore.java:367)
      	at org.jclouds.blobstore.internal.BaseBlobStore.putMultipartBlob(BaseBlobStore.java:347)
      	at org.jclouds.azureblob.blobstore.AzureBlobStore.putBlob(AzureBlobStore.java:238)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:37)
      	at com.sun.proxy.$Proxy45.putBlob(Unknown Source)
      	at org.abelsromero.jclouds.App.putContent(App.java:86)
      	at org.abelsromero.jclouds.App.testFile(App.java:47)
      	at org.abelsromero.jclouds.App.doStuff(App.java:34)
      	at org.abelsromero.jclouds.App.main(App.java:21)
      

      Here is a repo to reproduce the issue: https://github.com/abelsromero/jclouds-azure-streams. Beware that download is 170 MB in files.
      To run it, create a file called "config.properties" with this form under src/main/resources

      storage.account.name=
      storage.account.key=
      storage.container.name=
      

      To enable and disable multipart, change the value here https://github.com/abelsromero/jclouds-azure-streams/blob/master/src/main/java/org/abelsromero/jclouds/App.java#L33

      Right now we are considering disabling multipart on some scenarios and in other we are forced to create temporal files, so none of this are really good.

      2. Setting `jclouds.mpu.parts.size` to a different value than the default 32MB does not seem to do anything.

        Attachments

          Activity

            People

            • Assignee:
              gaul Andrew Gaul
              Reporter:
              Abel S.Romero Abel Salgado Romero
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: