[HADOOP-18458] AliyunOSS: AliyunOSSBlockOutputStream to support heap/off-heap buffer before uploading data to OSS

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.3, 3.1.4, 2.10.2, 3.2.4, 3.3.4
• Fix Version/s: 3.4.0
• Component/s: fs/oss
• Labels: pull-request-available
• Target Version/s: 3.4.0
• Hadoop Flags: Reviewed
• Assignee: wujinhu
• Reporter: wujinhu
• Created: 20/Sep/22 02:12
• Updated: 28/Jan/24 00:07
• Resolved: 29/Mar/23 08:04

Description

Recently, our customers raise a requirement: AliyunOSSBlockOutputStream should support heap/off-heap buffer before uploading data to OSS.

Currently, AliyunOSSBlockOutputStream buffers data in local directory before uploading to OSS, it is not efficient compared to memory.

Changes:

1. Adds heap/off-heap buffers
2. Adds limitation of memory used, and fallback to disk

Issue Links

• links to: GitHub Pull Request #4912