Uploaded image for project: 'jclouds'
  1. jclouds
  2. JCLOUDS-631

Slowdown of putBlob operation for the FileSystem provider

    XMLWordPrintableJSON

    Details

      Description

      The following (incomplete) piece of code when run for the filesystem provider and swift, for uploading a 1GB file shows 2-3x slowdown for the filesystem provider.

      Note that the code only writes through jclouds, there is no reading involved.

       public void sendLargeFilesOnlyJclouds() throws Exception
          {
              LOG.info("Working with {}", p);
              Environment env = Environment.builder().setName("SomeName").addProperties(p).build();
      
              String providerName = env.getConnectionModules().get(0).getName();
              String provider = checkNotNull(env.getProperty(providerName + ".provider"), new ICStoreException("No provider specified"));
      
              Iterable<? extends Module> modules =
                      ImmutableSet.of(
                              new SLF4JLoggingModule()
                      );
      
              if(provider.equals("s3"))
              {
                  provider = "aws-s3";
                  modules =
                          ImmutableSet.<Module>builder()
                                  .addAll(modules)
                                  .add(new ICStoreMultipartUploadModule())
                                  .build();
              }
              Properties properties = new Properties();
              // store provider credentials in properties
              properties.setProperty(provider + ".identity", env.getProperty( providerName + ".identity", ""));
              properties.setProperty(provider + ".credential", env.getProperty(providerName + ".credential", ""));
              properties.putAll(env.getPropertiesForModule(providerName));
      
              final BlobStoreContext blobStoreContext = ContextBuilder.newBuilder(provider)
                      .overrides(properties)
                      .modules(modules)
                      .buildView(BlobStoreContext.class);
      
              final BlobStore bs = blobStoreContext.getBlobStore();
      
              bs.createContainerInLocation(null, "mycontainer");
      
              for (int i = 0; i < num_invocations; i++)
              {
                  InputStream fis = new FakeInputStream(FAKE_IS_SEED, blobSize);
                  Blob blob = bs.blobBuilder("myblob").payload(fis).build();
      
                  LOG.info("Invocation number {}", i);
                  long startTime = System.currentTimeMillis();
                  bs.putBlob("mycontainer", blob);
                  long stopTime = System.currentTimeMillis();
      
                  LOG.info("Running time for one blob is {}", stopTime - startTime);
              }
          }
      

      Where FakeInputStream is an input stream that produces a deterministic output (though method read(), of a given size, and Environment is just a smarter container for Properties.

        Attachments

          Activity

            People

            • Assignee:
              gaul Andrew Gaul
              Reporter:
              knl Nikola Knezevic
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: