[JCLOUDS-626] OutOfMemoryError with FileSystem provider, large content from non-repeatable input streams

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.7.3
• Fix Version/s: 1.8.0
• Component/s: jclouds-blobstore
• Environment: Mac OS X 10.9.4, Oracle Java 1.7.0_55; RHEL 6.5, IBM Java
• Assignee: Andrew Gaul
• Reporter: Nikola Knezevic
• Created: 14/Jul/14 11:24
• Updated: 17/Jul/14 05:48
• Resolved: 16/Jul/14 15:41

Description

The following (incomplete) piece of code dies on my systems with a stacktrace (given below):

 public void sendLargeFilesOnlyJclouds() throws Exception
    {
        LOG.info("Working with {}", p);
        Environment env = Environment.builder().setName("SomeName").addProperties(p).build();

        String providerName = env.getConnectionModules().get(0).getName();
        String provider = checkNotNull(env.getProperty(providerName + ".provider"), new ICStoreException("No provider specified"));

        Iterable<? extends Module> modules =
                ImmutableSet.of(
                        new SLF4JLoggingModule()
                );

        if(provider.equals("s3"))
        {
            provider = "aws-s3";
            modules =
                    ImmutableSet.<Module>builder()
                            .addAll(modules)
                            .add(new ICStoreMultipartUploadModule())
                            .build();
        }
        Properties properties = new Properties();
        // store provider credentials in properties
        properties.setProperty(provider + ".identity", env.getProperty( providerName + ".identity", ""));
        properties.setProperty(provider + ".credential", env.getProperty(providerName + ".credential", ""));
        properties.putAll(env.getPropertiesForModule(providerName));

        final BlobStoreContext blobStoreContext = ContextBuilder.newBuilder(provider)
                .overrides(properties)
                .modules(modules)
                .buildView(BlobStoreContext.class);

        final BlobStore bs = blobStoreContext.getBlobStore();

        bs.createContainerInLocation(null, "mycontainer");

        for (int i = 0; i < num_invocations; i++)
        {
            InputStream fis = new FakeInputStream(FAKE_IS_SEED, blobSize);
            Blob blob = bs.blobBuilder("myblob").payload(fis).build();

            LOG.info("Invocation number {}", i);
            long startTime = System.currentTimeMillis();
            bs.putBlob("mycontainer", blob);
            long stopTime = System.currentTimeMillis();

            LOG.info("Running time for one blob is {}", stopTime - startTime);
        }
    }

Where FakeInputStream is an input stream that produces a deterministic output (though method read(), of a given size, and Environment is just a smarter container for Properties.

The resulting stack trace follows, and I get it during the first iteration:

2014-07-14@12:03:48.653 I            [main  ][        ] soTest:sendLargeFilesOnlyJclouds - Invocation number 0
java.lang.OutOfMemoryError: Requested array size exceeds VM limit
Dumping heap to java_pid8693.hprof ...
Heap dump file created [2145617933 bytes in 85.677 secs]
Exception in thread "main" java.lang.OutOfMemoryError: Requested array size exceeds VM limit
	at java.util.Arrays.copyOf(Arrays.java:2271)
	at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
	at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
	at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
	at com.google.common.io.ByteStreams.copy(ByteStreams.java:173)
	at com.google.common.io.ByteStreams.toByteArray(ByteStreams.java:214)
	at com.google.common.io.ByteSource.read(ByteSource.java:266)
	at com.google.common.io.ByteStreams.toByteArray(ByteStreams.java:226)
	at org.jclouds.filesystem.strategy.internal.FilesystemStorageStrategyImpl.putBlob(FilesystemStorageStrategyImpl.java:205)
	at org.jclouds.blobstore.LocalAsyncBlobStore.putBlob(LocalAsyncBlobStore.java:397)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:37)
	at com.sun.proxy.$Proxy45.putBlob(Unknown Source)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.google.common.reflect.Invokable$MethodInvokable.invokeInternal(Invokable.java:197)
	at com.google.common.reflect.Invokable.invoke(Invokable.java:102)
	at org.jclouds.rest.internal.InvokeAndCallGetOnFutures.apply(InvokeAndCallGetOnFutures.java:68)
	at org.jclouds.rest.internal.InvokeAndCallGetOnFutures.apply(InvokeAndCallGetOnFutures.java:40)
	at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
	at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
	at com.sun.proxy.$Proxy46.putBlob(Unknown Source)
	at com.ibm.icstore.stress.LargeFilesTest.sendLargeFilesOnlyJclouds(LargeFilesTest.java:153)
	at com.ibm.icstore.stress.LargeFilesTest.main(LargeFilesTest.java:163)

I suspect the problem is with the following piece of code in the method putBlob in FilesystemStorageStrategyImpl.java:

         else {
            if (!payload.isRepeatable()) {
               payload = Payloads.newPayload(ByteStreams.toByteArray(payload));
            }
            Files.copy(payload, outputFile);
         }

Judging by the previous fix for the issue (didn't fix it), the culprit seems to be:
Payloads.newPayload(ByteStreams.toByteArray(payload));

Why not completely ditch that line, and just go for Files.copy(), as it accepts an InputStream, and doesn't require a repeatable IS?