[JCLOUDS-1488] Filesystem list call with prefix is slow in large containers

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.1.1
• Fix Version/s: 2.2.0, 2.1.2
• Component/s: jclouds-blobstore
• Labels: filesystem
• Environment: Java version: java version "1.8.0_131" ; Operating system: Fedora 27 x86_64
• Assignee: Andrew Gaul
• Reporter: Lari Sinisalo
• Created: 18/Jan/19 10:11
• Updated: 22/Jan/23 09:31
• Resolved: 30/Jan/19 01:47

Description

When the filesystem blobstore is used, running the following code takes very long if there are a lot of files in the container:

    ListContainerOptions options = new ListContainerOptions();

    options.prefix("test-container-subdirectory/");

    Set<? extends StorageMetadata> results =
      blobStore.list("test-container",options);

See the attached Java source file JCLOUDS1488.java for the full code.

On my system, running the attached Java code takes over 10 seconds to list a single file if there are 500,000 files in the container outside that prefix.

Output from the attached code:

Number of blobs listed: 1
First listed blob: test-container-subdirectory/file-to-list
Time it took to list the blobs: 13256 ms

A more general version of this problem was reported previously in JCLOUDS-1371.