[HADOOP-14423] s3guard will set file length to -1 on a putObjectDirect(stream, -1) call

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.0.0-alpha4
• Fix Version/s: HADOOP-13345
• Component/s: fs/s3
• Target Version/s: HADOOP-13345
• Hadoop Flags: Reviewed
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 15/May/17 11:34
• Updated: 29/Jun/17 23:03
• Resolved: 29/Jun/17 23:03

Description

You can pass a negative number into S3AFileSystem.putObjectDirect, which means "put until the end of the stream". S3guard has been using this len argument: it needs to be using the actual number of bytes uploaded. Also relevant with client side encryption, when the amount of data put > the amount of data in the file or stream.

Noted in the committer branch after I added some more assertions, I've changed it there so making changes to S3AFS.putObjectDirect to pull the content length to pass in to finishedWrite() from the PutObjectResult instead. This can be picked into the s3guard branch

Issue Links

• depends upon: HADOOP-13998 Merge initial S3guard release into trunk
• Is contained by: HADOOP-13786 Add S3A committers for zero-rename commits to S3 endpoints