[HADOOP-13106] if fs.s3a.block.size option == 0, use partition size option for blocksize

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Won't Fix
• Affects Version/s: 2.7.2
• Component/s: fs/s3
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 06/May/16 11:04
• Updated: 13/Jun/16 10:16
• Resolved: 13/Jun/16 10:16

Description

Input perf is best if blocksize matches the partition size used to upload the data.

There's currently no way of knowing this —but S3A does know the partition size to use when uploading data.

I propose picking that up as the blocksize if the configured value of fs.s3a.block.size is zero. This value is utterly illegal today (more precisely, it will break any app using it to calculate splits), even when there is no check for its range