[HADOOP-15780] S3Guard: document how to deal with non-S3Guard processes writing data to S3Guarded buckets

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Target Version/s: 3.3.0
• Assignee: Gabor Bota
• Reporter: Aaron Fabbri
• Created: 21/Sep/18 00:46
• Updated: 29/Mar/19 14:00
• Resolved: 29/Mar/19 14:00

Description

Our general policy for S3Guard is this: All modifiers of a bucket that is configured for use with S3Guard, must use S3Guard. Otherwise, the MetadataStore will not be properly updated as the S3 bucket changes and problems will arise.

There are limited circumstances in which may be safe to have an external (non-s3guard) process writing data. There are also scenarios where it definitely breaks things.

I think we should start by documenting the cases that this works / does not work for. After we've enumerated that, we can suggest enhancements as needed to make this sort of configuration easier to use.

To get the ball rolling, some things that do not work:

• Deleting a path p with S3Guard, then writing a new file at path p without S3guard (will still have delete marker in S3Guard, making the file appear to be deleted but still visible in S3 due to false "eventual consistency") (as [e-mail] and I have discussed)
• When fs.s3a.metadatastore.authoritative is true, adding files to directories without S3Guard, then listing with S3Guard may exclude externally-written files from listings.

(Note, there are also S3A interop issues with other non-S3A clients even without S3Guard, due to the unique way S3A interprets empty directory markers).

Issue Links

• relates to: HADOOP-15999 S3Guard: Better support for out-of-band operations