[JCLOUDS-631] Slowdown of putBlob operation for the FileSystem provider

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Cannot Reproduce
• Affects Version/s: 1.8.0
• Component/s: jclouds-blobstore
• Labels: filesystem performance
• Assignee: Andrew Gaul
• Reporter: Nikola Knezevic
• Created: 17/Jul/14 06:08
• Updated: 24/Mar/15 22:58
• Resolved: 21/Jan/15 00:57

Description

The following (incomplete) piece of code when run for the filesystem provider and swift, for uploading a 1GB file shows 2-3x slowdown for the filesystem provider.

Note that the code only writes through jclouds, there is no reading involved.

 public void sendLargeFilesOnlyJclouds() throws Exception
    {
        LOG.info("Working with {}", p);
        Environment env = Environment.builder().setName("SomeName").addProperties(p).build();

        String providerName = env.getConnectionModules().get(0).getName();
        String provider = checkNotNull(env.getProperty(providerName + ".provider"), new ICStoreException("No provider specified"));

        Iterable<? extends Module> modules =
                ImmutableSet.of(
                        new SLF4JLoggingModule()
                );

        if(provider.equals("s3"))
        {
            provider = "aws-s3";
            modules =
                    ImmutableSet.<Module>builder()
                            .addAll(modules)
                            .add(new ICStoreMultipartUploadModule())
                            .build();
        }
        Properties properties = new Properties();
        // store provider credentials in properties
        properties.setProperty(provider + ".identity", env.getProperty( providerName + ".identity", ""));
        properties.setProperty(provider + ".credential", env.getProperty(providerName + ".credential", ""));
        properties.putAll(env.getPropertiesForModule(providerName));

        final BlobStoreContext blobStoreContext = ContextBuilder.newBuilder(provider)
                .overrides(properties)
                .modules(modules)
                .buildView(BlobStoreContext.class);

        final BlobStore bs = blobStoreContext.getBlobStore();

        bs.createContainerInLocation(null, "mycontainer");

        for (int i = 0; i < num_invocations; i++)
        {
            InputStream fis = new FakeInputStream(FAKE_IS_SEED, blobSize);
            Blob blob = bs.blobBuilder("myblob").payload(fis).build();

            LOG.info("Invocation number {}", i);
            long startTime = System.currentTimeMillis();
            bs.putBlob("mycontainer", blob);
            long stopTime = System.currentTimeMillis();

            LOG.info("Running time for one blob is {}", stopTime - startTime);
        }
    }

Where FakeInputStream is an input stream that produces a deterministic output (though method read(), of a given size, and Environment is just a smarter container for Properties.