[HADOOP-15625] S3A input stream to use etags/version number to detect changed source files

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.3.0, 3.2.1
• Component/s: fs/s3
• Target Version/s: 3.3.0
• Assignee: Ben Roling
• Reporter: Brahma Reddy Battula
• Created: 23/Jul/18 19:30
• Updated: 15/May/19 09:49
• Resolved: 14/Mar/19 19:48

Description

S3A input stream doesn't handle changing source files any better than the other cloud store connectors. Specifically: it doesn't noticed it has changed, caches the length from startup, and whenever a seek triggers a new GET, you may get one of: old data, new data, and even perhaps go from new data to old data due to eventual consistency.

We can't do anything to stop this, but we could detect changes by

1. caching the etag of the first HEAD/GET (we don't get that HEAD on open with S3Guard, BTW)
2. on future GET requests, verify the etag of the response
3. raise an IOE if the remote file changed during the read.

It's a more dramatic failure, but it stops changes silently corrupting things.

Issue Links

• is related to: HADOOP-16313 multipart/huge file upload tests to look at checksums returned
• is related to: HADOOP-15999 S3Guard: Better support for out-of-band operations
• is related to: HADOOP-16090 S3A Client to add explicit support for versioned stores
• is required by: HADOOP-15751 AWS Data read stack trace in S3a putObjectDirect
• relates to: HADOOP-16085 S3Guard: use object version or etags to protect against inconsistent read after replace/overwrite
• relates to: HADOOP-15894 getFileChecksum() needs to adopt S3Guard
• relates to: HADOOP-16202 Enhance openFile() for better read performance against object stores
• requires: HADOOP-15229 Add FileSystem builder-based openFile() API to match createFile(); S3A to implement S3 Select through this API.
• links to: GitHub Pull Request #602