[HDDS-9911] EC bucket Write Performance improve

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: EC
• Assignee: ChenXi
• Reporter: ChenXi
• Created: 13/Dec/23 07:19
• Updated: 15/Feb/24 08:53

Description

Improve write performance of EC type bucket. 

Improve the write performance of EC type bucket, about 5 ~ 7 times for small files, about 2 ~ 3 times for large files.

 

Single Threaded Client

4K Write QPS: about 70 -> 360

ozone freon ommg --operation CREATE_KEY -n 100000  -t 1 --volume=s3v --bucket=bucket-object-ec --duration 60 --size=4k

Before

After

 

 

MultiThreaded Client

4K Write QPS: about 800 -> 5500

ozone freon ommg --operation CREATE_KEY -n 100000  -t 50 --volume=s3v --bucket=bucket-object-ec --duration 60 --size=4k

Before

 

After

 

S3G

4K Write QPS: about 800 -> 5000

warp put --host localhost:9878 --obj.size=4KiB --duration=1m --concurrent=100 --keep-data --noclear --bucket=bucket-object-ec --access-key=ozone --secret-key=xxxxxxx

Issue Links

• incorporates: HDDS-9912 EC client Reusing thread resources

Sub-Tasks

• Reduce number of times configuration is loaded in Ozone client: Resolved
• EC client not allocate Buffer for 0 size key: Open