[HADOOP-18637] S3A to support upload of files greater than 2 GB using DiskBlocks

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0, 3.3.6
• Fix Version/s: 3.4.0, 3.3.6
• Component/s: fs/s3
• Labels: pull-request-available
• Target Version/s: 3.4.0, 3.3.6
• Hadoop Flags: Reviewed
• Assignee: Harshit Gupta
• Reporter: Harshit Gupta
• Created: 21/Feb/23 09:05
• Updated: 27/Jan/24 08:41
• Resolved: 27/Apr/23 10:01

Description

Use S3A Diskblocks to support the upload of files greater than 2 GB using DiskBlocks. Currently, the max upload size of a single block is ~2GB. 

cc: mthakur [e-mail] mehakmeet 

Issue Links

• causes: HADOOP-18738 Error message when multipart disabled and buffering is array not useful
• is related to: HADOOP-18695 S3A: reject multipart copy requests when disabled
• links to: GitHub Pull Request #5481
• links to: GitHub Pull Request #5543
• links to: GitHub Pull Request #5630
• links to: GitHub Pull Request #5641