[HDDS-9843] Ozone client high memory (heap) utilization

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.5.0
• Component/s: Ozone Client, S3, s3gateway
• Labels: Performance performance pull-request-available
• Assignee: Duong
• Reporter: Duong
• Created: 05/Dec/23 20:05
• Updated: 09/Feb/24 18:14
• Resolved: 09/Feb/24 18:05

Description

Ozone client high memory utilization during write - should use pooled direct buffer for chunks

Found this issue when working on S3.

S3G memory footprint is too high when reading/writing small objects.

Ran a test to write thousand of small (400 bytes) objects per seconds and memory utilization and gc is too high.

This is issue is not S3 specific, but happens on ozone-client in general.

Issue Links

• is blocked by: HDDS-10288 Checksum to support direct buffers
• is related to: HDDS-10043 java.lang.OutOfMemoryError on FSDataOutputStream write ops
• is related to: HDDS-10283 Apply zero copy to Ozone client reads chunks
• split from: HDDS-9536 Datanode perf: Copying (heap) buffers is costly
• links to: GitHub Pull Request #6153