Uploaded image for project: 'jclouds'
  1. jclouds
  2. JCLOUDS-1488

Filesystem list call with prefix is slow in large containers

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.1.1
    • Fix Version/s: 2.2.0, 2.1.2
    • Component/s: jclouds-blobstore
    • Labels:
    • Environment:
      Java version: java version "1.8.0_131"
      Operating system: Fedora 27 x86_64

      Description

      When the filesystem blobstore is used, running the following code takes very long if there are a lot of files in the container:

          ListContainerOptions options = new ListContainerOptions();
      
          options.prefix("test-container-subdirectory/");
      
          Set<? extends StorageMetadata> results =
            blobStore.list("test-container",options);
      

      See the attached Java source file JCLOUDS1488.java for the full code.

      On my system, running the attached Java code takes over 10 seconds to list a single file if there are 500,000 files in the container outside that prefix.

      Output from the attached code:

      Number of blobs listed: 1
      First listed blob: test-container-subdirectory/file-to-list
      Time it took to list the blobs: 13256 ms
      

      A more general version of this problem was reported previously in JCLOUDS-1371.

        Attachments

        1. JCLOUDS1488.java
          3 kB
          Lari Sinisalo

          Activity

            People

            • Assignee:
              gaul Andrew Gaul
              Reporter:
              Lari Sinisalo Lari Sinisalo
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: