Uploaded image for project: 'jclouds'
  1. jclouds
  2. JCLOUDS-1488

Filesystem list call with prefix is slow in large containers

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.1.1
    • 2.2.0, 2.1.2
    • jclouds-blobstore
    • Java version: java version "1.8.0_131"
      Operating system: Fedora 27 x86_64

    Description

      When the filesystem blobstore is used, running the following code takes very long if there are a lot of files in the container:

          ListContainerOptions options = new ListContainerOptions();
      
          options.prefix("test-container-subdirectory/");
      
          Set<? extends StorageMetadata> results =
            blobStore.list("test-container",options);
      

      See the attached Java source file JCLOUDS1488.java for the full code.

      On my system, running the attached Java code takes over 10 seconds to list a single file if there are 500,000 files in the container outside that prefix.

      Output from the attached code:

      Number of blobs listed: 1
      First listed blob: test-container-subdirectory/file-to-list
      Time it took to list the blobs: 13256 ms
      

      A more general version of this problem was reported previously in JCLOUDS-1371.

      Attachments

        1. JCLOUDS1488.java
          3 kB
          Lari Sinisalo

        Activity

          People

            gaul Andrew Gaul
            Lari Sinisalo Lari Sinisalo
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: