[HADOOP-16900] Very large files can be truncated when written through S3AFileSystem

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.1
• Fix Version/s: 3.3.1, 3.4.0
• Component/s: fs/s3
• Labels: s3
• Assignee: Mukund Thakur
• Reporter: Andrew Olson
• Created: 02/Mar/20 22:16
• Updated: 17/May/21 04:16
• Resolved: 20/May/20 12:45

Description

If a written file size exceeds 10,000 * fs.s3a.multipart.size, a corrupt truncation of the S3 object will occur as the maximum number of parts in a multipart upload is 10,000 as specified by the S3 API, and there is an apparent bug where this failure is not fatal allowing the multipart upload operation to be marked as successfully completed without being fully complete.

Issue Links

• links to: GitHub Pull Request #1944