[HDDS-8773] [S3G] Improve list performance in FSO bucket

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.3.0
• Fix Version/s: 1.4.0
• Component/s: S3
• Labels: pull-request-available
• Assignee: Hongbing Wang
• Reporter: Hongbing Wang
• Created: 07/Jun/23 05:23
• Updated: 09/Nov/23 04:13
• Resolved: 09/Nov/23 04:13

Description

Option --delimiter '/'  in ListObjectsV2  Is a commonly used option. Common scenarios are as follows:

• simple aws s3 cmd like aws s3 --endpoint http://<ip>:9878 ls buk/dir/
• aws s3api cmd like aws s3api --endpoint http://<ip>:9878 list-objects-v2 --bucket buk1-ln --prefix '' --delimiter '/'
• fuse mount s3 bucket and ls cmd

Only listing immediate children node of prefix is needed in the above scenario.

In the current implementation of FSO bucket, the object is listed by Depth-First-Search algorithm, and then filtered by delimiter, which greatly reduces the performance.

It was reduced from tens of seconds to 3 seconds in my test environment after optimization.

Issue Links

• links to: GitHub Pull Request #4868