Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Workaround
-
2.0.2
-
None
-
None
-
OSX, Linux (CentOS, Ubuntu)
Description
Obtain template code from: https://github.com/jclouds/jclouds-examples/tree/master/blobstore-largeblob
I've updated this file to specify region and had a bug (content length wasn't set). Attaching code to this JIRA.
Create a dummy file: mkfile -n 100g dummyFile.txt
Now try to upload this large file to a S3 bucket.
Tested scenarios:
8GB or 16GB heap does not change anything. As well, as indicated logs of swapping occurs during the upload. Peak memory used before it crashes is ~15GB.
Swap usage went from 200MB to 2GB between these two runs
2017-08-01 18:08:00,879 DEBUG [jclouds.blobstore] (main) 107374182400 bytes partitioned in 99 parts of part size: 1073741824, remaining: 1073741824
Scenario 1: 8GB Heap space + 4 transfer threads
java -Xmx8G -jar jclouds/blobstore-largeblob/target/blobstore-largeblob-jar-with-dependencies.jar aws-s3 IDENTITY "CREDENTIAL" dummyFile.txt us-west-2 CONTAINER_NAME dummyFile.txt
log4j:WARN No appenders could be found for logger (jclouds.signature).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Transferring file: dummyFile.txt size: 107374182400
Calling putBlob
Java heap space
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3332)
at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
at java.lang.StringBuilder.append(StringBuilder.java:136)
at org.jclouds.logging.internal.Wire.wire(Wire.java:65)
at org.jclouds.logging.internal.Wire.copy(Wire.java:96)
at org.jclouds.logging.internal.Wire.output(Wire.java:156)
at org.jclouds.logging.internal.Wire.output(Wire.java:133)
at org.jclouds.http.HttpUtils.wirePayloadIfEnabled(HttpUtils.java:288)
at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:89)
at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
at com.sun.proxy.$Proxy42.uploadPart(Unknown Source)
at org.jclouds.s3.blobstore.strategy.internal.SequentialMultipartUploadStrategy.prepareUploadPart(SequentialMultipartUploadStrategy.java:111)
at org.jclouds.s3.blobstore.strategy.internal.SequentialMultipartUploadStrategy.execute(SequentialMultipartUploadStrategy.java:93)
at org.jclouds.aws.s3.blobstore.AWSS3BlobStore.putBlob(AWSS3BlobStore.java:87)
at org.jclouds.examples.blobstore.largeblob.MainApp.main(MainApp.java:162)
—
Scenario 2: 16GB Heap space + 4 transfer threads
l-mp-udhansingh:~ udhansingh$ java -Xmx16G -jar jclouds/blobstore-largeblob/target/blobstore-largeblob-jar-with-dependencies.jar aws-s3 IDENTITY "CREDENTIAL" dummyFile.txt us-west-2 CONTAINER_NAME dummyFile.txt
log4j:WARN No appenders could be found for logger (jclouds.signature).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Transferring file: dummyFile.txt size: 107374182400
Calling putBlob
null
java.lang.OutOfMemoryError
at java.lang.AbstractStringBuilder.hugeCapacity(AbstractStringBuilder.java:161)
at java.lang.AbstractStringBuilder.newCapacity(AbstractStringBuilder.java:155)
at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:125)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
at java.lang.StringBuilder.append(StringBuilder.java:136)
at org.jclouds.logging.internal.Wire.wire(Wire.java:65)
at org.jclouds.logging.internal.Wire.copy(Wire.java:96)
at org.jclouds.logging.internal.Wire.output(Wire.java:156)
at org.jclouds.logging.internal.Wire.output(Wire.java:133)
at org.jclouds.http.HttpUtils.wirePayloadIfEnabled(HttpUtils.java:288)
at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:89)
at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
at com.sun.proxy.$Proxy42.uploadPart(Unknown Source)
at org.jclouds.s3.blobstore.strategy.internal.SequentialMultipartUploadStrategy.prepareUploadPart(SequentialMultipartUploadStrategy.java:111)
at org.jclouds.s3.blobstore.strategy.internal.SequentialMultipartUploadStrategy.execute(SequentialMultipartUploadStrategy.java:93)
at org.jclouds.aws.s3.blobstore.AWSS3BlobStore.putBlob(AWSS3BlobStore.java:87)
at org.jclouds.examples.blobstore.largeblob.MainApp.main(MainApp.java:162)
—
Scenario 3: 16GB Heap space + 1 transfer thread
java -Xmx16G -jar jclouds/blobstore-largeblob/target/blobstore-largeblob-jar-with-dependencies.jar aws-s3 IDENTITY "CREDENTIAL" dummyFile.txt us-west-2 CONTAINER_NAME dummyFile.txt plainhttp 1
log4j:WARN No appenders could be found for logger (jclouds.signature).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Transferring file: dummyFile.txt size: 107374182400
Calling putBlob
null
java.lang.OutOfMemoryError
at java.lang.AbstractStringBuilder.hugeCapacity(AbstractStringBuilder.java:161)
at java.lang.AbstractStringBuilder.newCapacity(AbstractStringBuilder.java:155)
at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:125)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
at java.lang.StringBuilder.append(StringBuilder.java:136)
at org.jclouds.logging.internal.Wire.wire(Wire.java:65)
at org.jclouds.logging.internal.Wire.copy(Wire.java:96)
at org.jclouds.logging.internal.Wire.output(Wire.java:156)
at org.jclouds.logging.internal.Wire.output(Wire.java:133)
at org.jclouds.http.HttpUtils.wirePayloadIfEnabled(HttpUtils.java:288)
at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:89)
at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
at com.sun.proxy.$Proxy42.uploadPart(Unknown Source)
at org.jclouds.s3.blobstore.strategy.internal.SequentialMultipartUploadStrategy.prepareUploadPart(SequentialMultipartUploadStrategy.java:111)
at org.jclouds.s3.blobstore.strategy.internal.SequentialMultipartUploadStrategy.execute(SequentialMultipartUploadStrategy.java:93)
at org.jclouds.aws.s3.blobstore.AWSS3BlobStore.putBlob(AWSS3BlobStore.java:87)
at org.jclouds.examples.blobstore.largeblob.MainApp.main(MainApp.java:162)
Scenario 4: 16GB Heap space + 4 transfer thread ; using file input stream (instead of byte stream)
java -Xmx16G -jar git/bitbucket/know-how/java/jclouds/blobstore-largeblob/target/blobstore-largeblob-jar-with-dependencies.jar aws-s3 IDENTITY "CREDENTIAL" dummyFile.txt us-west-2 CONTAINER_NAME dummyFile.txt
log4j:WARN No appenders could be found for logger (jclouds.signature).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Transferring file: dummyFile.txt size: 107374182400
Calling putBlob
null
java.lang.OutOfMemoryError
at java.lang.AbstractStringBuilder.hugeCapacity(AbstractStringBuilder.java:161)
at java.lang.AbstractStringBuilder.newCapacity(AbstractStringBuilder.java:155)
at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:125)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
at java.lang.StringBuilder.append(StringBuilder.java:136)
at org.jclouds.logging.internal.Wire.wire(Wire.java:65)
at org.jclouds.logging.internal.Wire.output(Wire.java:184)
at org.jclouds.logging.internal.Wire.output(Wire.java:158)
at org.jclouds.logging.internal.Wire.output(Wire.java:131)
at org.jclouds.http.HttpUtils.wirePayloadIfEnabled(HttpUtils.java:288)
at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:89)
at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
at com.sun.proxy.$Proxy42.uploadPart(Unknown Source)
at org.jclouds.s3.blobstore.strategy.internal.SequentialMultipartUploadStrategy.prepareUploadPart(SequentialMultipartUploadStrategy.java:111)
at org.jclouds.s3.blobstore.strategy.internal.SequentialMultipartUploadStrategy.execute(SequentialMultipartUploadStrategy.java:93)
at org.jclouds.aws.s3.blobstore.AWSS3BlobStore.putBlob(AWSS3BlobStore.java:87)
at org.jclouds.examples.blobstore.largeblob.MainApp.main(MainApp.java:167)
Looks like the way JClouds construct the payload (using String Builder seems to cause an Out-of-memory issue.