[JCLOUDS-1366] OutOfMemory when InputStream referencing to big file is used as payload

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.0.0, 2.0.3
• Fix Version/s: 2.2.0, 2.1.2
• Component/s: jclouds-blobstore
• Environment: Linux and Windows
• Assignee: Andrew Gaul
• Reporter: Deyan
• Created: 13/Dec/17 16:39
• Updated: 04/Jan/19 23:45
• Resolved: 04/Jan/19 23:45

Description

If I use InputStream which source is large file (lets say 3GB) I am getting OOE. This is with default java VM options.
Here is the code I am using to construct the blob:

 File bigFile = new File(file);
 try (InputStream inputStream = new FileInputStream(f)) {
                Blob b = blobStore.blobBuilder(blobName)
                        .payload(inputStream).contentLength(f.length())
                        .contentDisposition(blobName)
                        .contentType(
                                MediaType.OCTET_STREAM)
                        .userMetadata(ImmutableMap.of("a", "b", "test", "beta"))
                        .build();
                blobStore.putBlob("test", bbbbb, multipart());
}

Stacktrace:

java.lang.OutOfMemoryError: Java heap space
	at org.jclouds.io.internal.BasePayloadSlicer$InputStreamPayloadIterator.getNextPayload(BasePayloadSlicer.java:101)
	at org.jclouds.io.internal.BasePayloadSlicer$InputStreamPayloadIterator.next(BasePayloadSlicer.java:90)
	at org.jclouds.io.internal.BasePayloadSlicer$InputStreamPayloadIterator.next(BasePayloadSlicer.java:63)
	at org.jclouds.blobstore.internal.BaseBlobStore.putMultipartBlob(BaseBlobStore.java:363)
	at org.jclouds.blobstore.internal.BaseBlobStore.putMultipartBlob(BaseBlobStore.java:349)
	at org.jclouds.s3.blobstore.S3BlobStore.putBlob(S3BlobStore.java:262)

If 'bigFile' is used as payload the bug is not reproducible.

Issue Links

• is related to: JCLOUDS-814 InputStreamPayloadIterator.getNextPayload buffers entire part in-memory