[HOP-3474] s3 is slower than it should be

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Migrated to GHI
• Component/s: VFS
• Assignee: Unassigned
• Reporter: Hans Van Akelyen
• Created: 01/Nov/21 15:43
• Updated: 16/Mar/23 15:35
• Resolved: 16/Mar/23 15:35

Description

It seems that we are losing precious time somewhere in VFS when listing objects from S3.

Further investigation is needed, attached is a sample pipeline to list items via the AWS SDK v1,v2 and using get file names.

The get file names transform is much slower. see if it is the VFS driver, or the transform.

But as also the browser is much slower than listing the files via the SDK I think something is wrong in the VFS driver.

Issue Links

• is related to: HOP-2988 Explorer Perspective: load folders lazily
• is related to: HOP-3574 Implement lazy loading for file browsing