Uploaded image for project: 'jclouds'
  1. jclouds
  2. JCLOUDS-1326

Unable to upload large objects to S3 using JClouds putBlob API

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Workaround
    • 2.0.2
    • None
    • jclouds-blobstore
    • None
    • OSX, Linux (CentOS, Ubuntu)

    Description

      Obtain template code from: https://github.com/jclouds/jclouds-examples/tree/master/blobstore-largeblob

      I've updated this file to specify region and had a bug (content length wasn't set). Attaching code to this JIRA.

      Create a dummy file: mkfile -n 100g dummyFile.txt

      Now try to upload this large file to a S3 bucket.

      Tested scenarios:
      8GB or 16GB heap does not change anything. As well, as indicated logs of swapping occurs during the upload. Peak memory used before it crashes is ~15GB.

      Swap usage went from 200MB to 2GB between these two runs

      2017-08-01 18:08:00,879 DEBUG [jclouds.blobstore] (main) 107374182400 bytes partitioned in 99 parts of part size: 1073741824, remaining: 1073741824

      Scenario 1: 8GB Heap space + 4 transfer threads
      java -Xmx8G -jar jclouds/blobstore-largeblob/target/blobstore-largeblob-jar-with-dependencies.jar aws-s3 IDENTITY "CREDENTIAL" dummyFile.txt us-west-2 CONTAINER_NAME dummyFile.txt
      log4j:WARN No appenders could be found for logger (jclouds.signature).
      log4j:WARN Please initialize the log4j system properly.
      log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
      Transferring file: dummyFile.txt size: 107374182400
      Calling putBlob
      Java heap space
      java.lang.OutOfMemoryError: Java heap space
      at java.util.Arrays.copyOf(Arrays.java:3332)
      at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
      at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
      at java.lang.StringBuilder.append(StringBuilder.java:136)
      at org.jclouds.logging.internal.Wire.wire(Wire.java:65)
      at org.jclouds.logging.internal.Wire.copy(Wire.java:96)
      at org.jclouds.logging.internal.Wire.output(Wire.java:156)
      at org.jclouds.logging.internal.Wire.output(Wire.java:133)
      at org.jclouds.http.HttpUtils.wirePayloadIfEnabled(HttpUtils.java:288)
      at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:89)
      at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
      at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
      at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
      at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
      at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
      at com.sun.proxy.$Proxy42.uploadPart(Unknown Source)
      at org.jclouds.s3.blobstore.strategy.internal.SequentialMultipartUploadStrategy.prepareUploadPart(SequentialMultipartUploadStrategy.java:111)
      at org.jclouds.s3.blobstore.strategy.internal.SequentialMultipartUploadStrategy.execute(SequentialMultipartUploadStrategy.java:93)
      at org.jclouds.aws.s3.blobstore.AWSS3BlobStore.putBlob(AWSS3BlobStore.java:87)
      at org.jclouds.examples.blobstore.largeblob.MainApp.main(MainApp.java:162)

      —

      Scenario 2: 16GB Heap space + 4 transfer threads

      l-mp-udhansingh:~ udhansingh$ java -Xmx16G -jar jclouds/blobstore-largeblob/target/blobstore-largeblob-jar-with-dependencies.jar aws-s3 IDENTITY "CREDENTIAL" dummyFile.txt us-west-2 CONTAINER_NAME dummyFile.txt
      log4j:WARN No appenders could be found for logger (jclouds.signature).
      log4j:WARN Please initialize the log4j system properly.
      log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
      Transferring file: dummyFile.txt size: 107374182400
      Calling putBlob
      null
      java.lang.OutOfMemoryError
      at java.lang.AbstractStringBuilder.hugeCapacity(AbstractStringBuilder.java:161)
      at java.lang.AbstractStringBuilder.newCapacity(AbstractStringBuilder.java:155)
      at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:125)
      at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
      at java.lang.StringBuilder.append(StringBuilder.java:136)
      at org.jclouds.logging.internal.Wire.wire(Wire.java:65)
      at org.jclouds.logging.internal.Wire.copy(Wire.java:96)
      at org.jclouds.logging.internal.Wire.output(Wire.java:156)
      at org.jclouds.logging.internal.Wire.output(Wire.java:133)
      at org.jclouds.http.HttpUtils.wirePayloadIfEnabled(HttpUtils.java:288)
      at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:89)
      at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
      at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
      at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
      at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
      at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
      at com.sun.proxy.$Proxy42.uploadPart(Unknown Source)
      at org.jclouds.s3.blobstore.strategy.internal.SequentialMultipartUploadStrategy.prepareUploadPart(SequentialMultipartUploadStrategy.java:111)
      at org.jclouds.s3.blobstore.strategy.internal.SequentialMultipartUploadStrategy.execute(SequentialMultipartUploadStrategy.java:93)
      at org.jclouds.aws.s3.blobstore.AWSS3BlobStore.putBlob(AWSS3BlobStore.java:87)
      at org.jclouds.examples.blobstore.largeblob.MainApp.main(MainApp.java:162)

      —

      Scenario 3: 16GB Heap space + 1 transfer thread

      java -Xmx16G -jar jclouds/blobstore-largeblob/target/blobstore-largeblob-jar-with-dependencies.jar aws-s3 IDENTITY "CREDENTIAL" dummyFile.txt us-west-2 CONTAINER_NAME dummyFile.txt plainhttp 1
      log4j:WARN No appenders could be found for logger (jclouds.signature).
      log4j:WARN Please initialize the log4j system properly.
      log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
      Transferring file: dummyFile.txt size: 107374182400
      Calling putBlob
      null
      java.lang.OutOfMemoryError
      at java.lang.AbstractStringBuilder.hugeCapacity(AbstractStringBuilder.java:161)
      at java.lang.AbstractStringBuilder.newCapacity(AbstractStringBuilder.java:155)
      at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:125)
      at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
      at java.lang.StringBuilder.append(StringBuilder.java:136)
      at org.jclouds.logging.internal.Wire.wire(Wire.java:65)
      at org.jclouds.logging.internal.Wire.copy(Wire.java:96)
      at org.jclouds.logging.internal.Wire.output(Wire.java:156)
      at org.jclouds.logging.internal.Wire.output(Wire.java:133)
      at org.jclouds.http.HttpUtils.wirePayloadIfEnabled(HttpUtils.java:288)
      at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:89)
      at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
      at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
      at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
      at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
      at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
      at com.sun.proxy.$Proxy42.uploadPart(Unknown Source)
      at org.jclouds.s3.blobstore.strategy.internal.SequentialMultipartUploadStrategy.prepareUploadPart(SequentialMultipartUploadStrategy.java:111)
      at org.jclouds.s3.blobstore.strategy.internal.SequentialMultipartUploadStrategy.execute(SequentialMultipartUploadStrategy.java:93)
      at org.jclouds.aws.s3.blobstore.AWSS3BlobStore.putBlob(AWSS3BlobStore.java:87)
      at org.jclouds.examples.blobstore.largeblob.MainApp.main(MainApp.java:162)

      Scenario 4: 16GB Heap space + 4 transfer thread ; using file input stream (instead of byte stream)

      java -Xmx16G -jar git/bitbucket/know-how/java/jclouds/blobstore-largeblob/target/blobstore-largeblob-jar-with-dependencies.jar aws-s3 IDENTITY "CREDENTIAL" dummyFile.txt us-west-2 CONTAINER_NAME dummyFile.txt
      log4j:WARN No appenders could be found for logger (jclouds.signature).
      log4j:WARN Please initialize the log4j system properly.
      log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
      Transferring file: dummyFile.txt size: 107374182400
      Calling putBlob
      null
      java.lang.OutOfMemoryError
      at java.lang.AbstractStringBuilder.hugeCapacity(AbstractStringBuilder.java:161)
      at java.lang.AbstractStringBuilder.newCapacity(AbstractStringBuilder.java:155)
      at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:125)
      at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
      at java.lang.StringBuilder.append(StringBuilder.java:136)
      at org.jclouds.logging.internal.Wire.wire(Wire.java:65)
      at org.jclouds.logging.internal.Wire.output(Wire.java:184)
      at org.jclouds.logging.internal.Wire.output(Wire.java:158)
      at org.jclouds.logging.internal.Wire.output(Wire.java:131)
      at org.jclouds.http.HttpUtils.wirePayloadIfEnabled(HttpUtils.java:288)
      at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:89)
      at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
      at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
      at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
      at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
      at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
      at com.sun.proxy.$Proxy42.uploadPart(Unknown Source)
      at org.jclouds.s3.blobstore.strategy.internal.SequentialMultipartUploadStrategy.prepareUploadPart(SequentialMultipartUploadStrategy.java:111)
      at org.jclouds.s3.blobstore.strategy.internal.SequentialMultipartUploadStrategy.execute(SequentialMultipartUploadStrategy.java:93)
      at org.jclouds.aws.s3.blobstore.AWSS3BlobStore.putBlob(AWSS3BlobStore.java:87)
      at org.jclouds.examples.blobstore.largeblob.MainApp.main(MainApp.java:167)

      Looks like the way JClouds construct the payload (using String Builder seems to cause an Out-of-memory issue.

      Attachments

        1. blobstore-largeblob.tar.gz
          6 kB
          Udayakumar Dhansingh

        Activity

          People

            gaul Andrew Gaul
            udhansingh Udayakumar Dhansingh
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: